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Foreword 


The European Investment Bank (EIB) has been investing in Africa since 1963. In 2020, it stepped up 
its investments to help African partners respond to the COVID-19 health and economic crisis. The EIB 
provided €5 billion for new private and public investment across the continent in 2020 — setting a new 
record for the Bank’s annual engagement. This finance will back more than €12 billion in investments 
in 28 African countries, with 71% of the funding benefiting fragile states and the least developed 
economies. The support will help African countries deal with the immediate health emergency and 
mitigate the epidemic, as well as address the economic effects of the crisis and progress towards 
achieving the Sustainable Development Goals. Projects signed in Africa in 20201 are expected to 
contribute to 210 million people receiving COVID-19 vaccines, 595 400 households being supplied with 
newly generated energy, 778 000 people enjoying improved water supply, and farmers benefiting from 
26 500 of hectares of newly irrigated land and 3 076 hectares of newly planted forest. 


The EIB partners private and public sector clients and uses innovative impact finance to support high- 
impact projects in Africa. As in previous years, around 50% of EIB lending in Africa in 2020 involved 
private sector partners. The EIB uses a window of the Africa, Caribbean and Pacific Investment Facility, 
the Impact Financing Envelope, to support higher-risk private sector initiatives with a big impact for 
poorer populations. 


As the European Union’s development bank, the EIB places the measurement of development 
impact at the very core of its business approach. Its Additionality and Impact Measurement 
Framework (AIM) captures evidence on the results of every EIB investment. The EIB aims to be at the 
forefront of rigorous impact assessment methodologies, combining information collected by the AIM 
with micro- and macro-level impact studies and independent evaluations. This creates a strong 
evidence base for gauging our development impact. Because we are accountable to our stakeholders, 
measuring our impact is essential. It also helps us and our clients learn what works, so that we can 
continually enhance our development impact. This learning process is particularly crucial for innovative 
approaches and projects, such as the initiatives supported under the Impact Financing Envelope. 


The Global Development Network (GDN) and the EIB have partnered on an impact research 
programme which has deepened the EIB’s insight into its contribution to sustainable and inclusive 
development in Africa and the Caribbean. The programme produced 16 impact studies for 
investments funded under the Impact Financing Envelope, in sectors ranging from healthcare to 
microfinance in eight African and Caribbean countries. The studies met high standards of technical 
quality and academic rigour, as confirmed by a strong and independent scientific committee. 


The EIB-GDN programme worked with local researchers and helped them develop their skills, while 
drawing on the technical know-how of top international experts. Under the programme, 30 talented 
researchers from Africa and the Caribbean spent a year developing an impact research study for an 
EIB-supported project. During this time, they were trained, guided and mentored by a scientific 
committee of global experts in impact evaluation. With top-level academics and policymakers from 
institutions including the Paris School of Economics, the World Bank and the International Initiative on 
Impact Evaluation (3ie), the scientific committee was put in place to guarantee the quality and 
credibility of the research produced under the programme. 


12020 Annual Report on EIB activity in the African, Caribbean and Pacific States and the Overseas Countries and Territories 
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The EIB will continue to invest in understanding our development impact as we work to build back 
better in the wake of the COVID-19 crisis, building on the lessons we have learnt from the EIB-GDN 
programme. The financing constraints faced by many of our partners and the high expectations of our 
EU stakeholders and citizens make it more important than ever that we support the investments 
capable of making the biggest difference on the ground. We are looking forward to continuing this 
research stream and leveraging the knowledge we generate to boost our accountability, our decision- 
making and the overall effectiveness of our development impact. 


Werner Hoyer 
President 
European Investment Bank 
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Introduction 


Authors: Pierre Jacquet and Debora Revoltella 


This book reports lessons learnt through a three-year partnership between the European Investment 
Bank (EIB) and the Global Development Network (GDN). The EIB-GDN Programme in Applied 
Development Finance applied rigorous evaluation techniques to ongoing private investments in 
developing countries, while building local research capacity. Local researchers provided an 
understanding of local contexts. The programme created a pool of local researchers with experience 
in development impact assessment. 


The book was co-edited by Pierre Jacquet, Emmanuel Jimenez, Shahrokh Fardoust and Alexandros 
Sarris; Matt Ripley was a major contributor; Francois Bourguignon, Arianna Legovini, Claudio Cali 
and Nina Fenton were co-authors; Alexis Atlani, Abhay Gupta and Brendan Harnoys-Vannier 
provided input. The authors based the chapters on their own experience of the programme and on 
feedback provided by the researchers, technical experts, private-sector funds and companies involved. 


The EIB-GDN Programme in Applied Development Finance started in 2017. Competitively selected 
young researchers from Africa and the Caribbean were asked to assess the impact of a set of projects 
financed under the EIB’s Impact Financing Envelope. Each cohort of ten researchers was matched 
with expert advisors who mentored them and assigned particular EIB investments to study. The 
researchers usually worked in pairs. The research outputs are referred to as deep dives. Deep dives 
are innovative research-based impact studies that collect data and use rigorous methods to investigate 
the nature of development impact and discuss causality. They provide an evaluative product that 
occupies the middle ground between descriptive studies focusing on outputs and much heavier, 
comprehensive scientific impact evaluations. During the three-year programme, 30 researchers 
produced 16 deep dives (EIB-GDN 2019, 2020, 2021). 


The programme innovated in three major ways: 


It used academic research techniques to study the development impact of ongoing private-sector 
investments. Measuring this impact requires unusual coordination between researchers and the 
private sector. Deep dives answer questions that interest funders and companies as well as academic 
researchers. Deep dives can fill a research gap between descriptive studies and scientific impact 
evaluations. 


It built capacity for evaluative research in developing countries. The programme relied on developing 
country researchers and mentored them to produce the deep dives. Researchers with an 
understanding of the local context of interventions add to the relevance, legitimacy and credibility of 
impact studies. The approach included local stakeholders in well-informed debates on how to promote 
economic and social development, and contribute to the sustainability of development impact. 
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Capacity building in itself added a development impact as compared with alternative ways to conduct 
such evaluative work. 


It offered an opportunity to boost accountability and development effectiveness. The programme 
complemented the Bank’s established results measurement framework by developing in-depth impact 
studies. The EIB gained new insights into the impact of ongoing projects. In some cases, these insights 
pointed towards ways to improve development impact management. Discussions with research teams 
allowed project promoters to reflect on their impact theses and on ways to boost impact alongside 
financial sustainability. The reputable and independent Expert Advisor Committee endorsed the 
impact studies, strengthening the accountability of EIB activities. 


This book describes the three-year journey taken by the organisations partnering under this 
programme, detailing the lessons learnt and opportunities for future work. 


Chapter 1 was co-authored by Shahrokh Fardoust, Pierre Jacquet and Matt Ripley. The chapter 
describes the role of impact investing in the development financing context, and the crucial role of 
impact measurement within this field. It illustrates the major issues and challenges behind impact 
measurement and the diversity of measurement practices. It discusses factors motivating 
measurement and describes a research gap in impact studies. It introduces deep dives as a way to 
bridge the research gap. 


Chapter 2 was led by Emmanuel Jimenez and co-authored by Pierre Jacquet, Alexandros Sarris, 
Shahrokh Fardoust, Nina Fenton, Claudio Cali, François Bourguigon and Arianna Legovini. The 
chapter comprehensively discusses the process behind the deep dives. It explains how research fellows 
were selected and paired, how projects were allocated and how mentoring was provided by expert 
advisors. It describes how deep dives were developed through an interactive and iterative process; the 
role of expert advisors in the choice of methods; and the challenges of data collection. It summarises 
the results, describes the process of building research capacity, and discusses what the research 
fellows learnt from the exercise. 


Chapter 3 was written by Matt Ripley. This chapter assesses the use of deep dives by stakeholders. It 
proposes an analytical framework based on four distinct underlying motivations for the use of deep 
dives: instrumental, conceptual, symbolic, and capacity building. It uses the framework to review the 
use of deep dives, based on a survey and discussions with key stakeholders. It discusses how deep 
dives were used along each of these four dimensions by the EIB and its private-sector partners, by 
introducing rigour in impact assessments while serving the interests of diverse stakeholders. 


Overall, the programme showed that tensions between the needs and interests of different 
stakeholders can be managed without compromising on quality. The resulting research-based deep 
dives can inform debates about development impact. The following chapters detail the main 
conclusions. Our key take-home messages are summarised below. 


Interactivity 


Interaction among the researchers, expert advisors, the EIB and investees shaped the deep dives. 
The researchers had to understand the context and the project, and test their initial ideas with expert 
advisors and then investees. The researchers needed the investees’ cooperation and identified 
research questions that provoked their interest. Communication was often challenging — between the 
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researchers and the private-sector companies they worked with and between the researchers and the 
ultimate readers of the deep dives. Deep dives mobilise academic approaches to address real-life and 
real-time issues, but they must be relevant and clearly communicated if they are to successfully engage 
private-sector companies and reach wider audiences. The researchers had to learn to combine 
academic rigour with appealing formulation, style and content. Substantive communication training 
and a significant research translation effort by EIB staff helped to achieve this aim. 


Knowing and understanding stakeholders’ interests and needs was key. Deep dive studies addressed 
these interests and needs where possible. Interaction was needed because the interests of 
stakeholders are diverse and research design must consider more than one set of interests. The EIB- 
GDN programme demonstrated that this interaction produced interesting research outputs, or deep 
dives, which were useful to stakeholders, including private-sector partners. 


Adaptability 


The Private-sector companies targeted by the studies often wanted insights with relevance to their 
business objectives as well as their development impact. Rapid results were needed as a relevant 
basis for operational decision-making. The researchers wanted to produce high-quality academic 
papers to further their careers. Their mentors, seasoned scholars, placed a strong emphasis on 
academic rigour and relevance, and required high-quality work based on methodologies tailored to 
the questions selected and the available data. The EIB-GDN programme required a balance between 
rigour, timeliness and relevance. Achieving this balance required constructive interaction among 
stakeholders, and adaptability in setting research questions and methodologies. This was a major 
learning point for the researchers. 


Data availability constrained the methods that could be applied and demanded adaptability. Data 
collection was strongly affected by the COVID-19 pandemic. The experience showed that a research- 
based approach can give useful and rigorous insights even when data requirements are met 
imperfectly. New data sources and innovative techniques (such as big data and machine learning) 
might help close future data gaps. 


Building capacity 


Substantial capacity was built among the research fellows who participated. Technical skills were 
boosted through interaction with global experts in impact evaluation. The researchers also gained a 
better understanding of the challenges of working in a non-academic context, discussing plans with 
non-researchers, reformulating feasible questions and approaches, and securing the buy-in of 
companies. The expert advisors were crucial in helping the young researchers to reorient their 
questions, methodological choices and presentations to meet private companies’ interests without 
compromising on academic quality. The researchers also benefited from supervision by EIB team 
members. This was particularly valuable in shaping communications to a non-academic context, 
achieving the best use of data collected and preparing executive summaries for companies. The GDN 
provided organisational and logistical support for facilitating interaction, formalising commitments, 
monitoring work advancement, organising annual seminars for new inductees and providing research 
communication training. 
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Future perspectives 


We see several ways in which this pilot exercise might usefully be scaled up. It could be expanded by 
increasing the number of impact studies conducted on private-sector projects that are expected to 
produce development impact. It could also be “deepened”, for example by choosing several projects 
in the same sector, so that the learning produced by impact studies could acquire a form of sector- 
based validity. 


Funding remains a challenge. Private impact investors are unlikely to fully cover the costs of partnering 
with academia to more rigorously assess their impact, because of the public-good nature of this 
research: the learning function of impact assessments benefits everyone. Impact studies — an exercise 
in mobilising rigorous methods to assess impact — are an example of how public money can support 
good research documenting the impact of private-sector projects. They demonstrate how new kinds 
of public-private partnerships can emerge. We hope that other development institutions will be 
inspired by the example of the EIB-funded programme and help to scale up the approach. 
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Summary and key messages 


This chapter examines impact investing — investment in private companies to generate socially 
desirable results in addition to financial returns — and the assessment of its effectiveness. Impact 
investing is growing fast, reflecting rising interest from private investors in their contribution to societal 
goals. Impact investments have become increasingly important both in responding to societal 
demands and in contributing to the investments needed for meeting the Sustainable Development 
Goals (SDGs), when other sources of financing fall short of the massive financing required. 


The growth of impact investing is focusing attention on impact measurement and the need for 
credible impact studies. Generating impact mobilises a chain of stakeholders — funders, intermediaries 
and investees — and knowledge brokers who monitor, measure and guide processes. Each stakeholder 
has their own definition of, interest in, and contribution to impact. The private sector, which focuses 
on profitability, would also typically consider potential trade-offs between financial and societal 
returns. The empirical jury is still out on the exact nature of such trade-offs. However, there are various 
types of investor within the private sector, not all of which focus on competitive financial returns. This 
leads to the possibility of blended finance in which, for example, public money motivates private 
investors, under the promise of a commercial return, towards producing public goods. 


Monitoring and evaluating the impact of impact investing involves significant challenges in defining 
a logical chain from inputs in an activity to outputs, outcomes and impacts. Measuring impact is 
complex and multidimensional, and discussing causality and attribution requires time, resources and 
capacity. There is no single optimal method: the choice of an appropriate method depends on 
resources, capacity and data availability. In practice, impact investors use a variety of approaches, try 
to improve clarity and credibility by adopting common standards and frameworks, and increasingly 
manage impact. However, there remains a research gap across investment studies. 


European Investment Bank (EIB) and Global Development Network (GDN) deep dives are impact 
studies that try to make research-based approaches more palatable for all stakeholders and fill a 
gap between impact reports and fully fledged scientific impact evaluations. They combine 
methodological rigour with the time and operational constraints of the impact investing stakeholders, 
especially investees who produce goods and services generating impact. 


Three main messages emerge from this chapter. First, impact studies mean different things to 
different stakeholders. There is a gap between the private value of impact studies (which focus on 
accountability, communication and targeted business-oriented learning) and their social value (which 
relies on the knowledge created about impact and how it can be generated). Bridging that gap requires 
analytical efforts, external incentives and resources. Second, rigorous academic research is important 
at all stages of the investment process. Evaluations should continue to consider a range of approaches 
and models, and evaluators should cultivate stronger connections with academia. Third, increasing 
cooperation with academia for impact assessments has a cost that is unlikely to be covered by impact 
investors themselves. To support the market for impact investing and social returns, efforts are needed 
to promote research-based evaluations that insist on rigour and control of biases, while considering 
private stakeholders’ interests and motivations. 
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This chapter examines impact investing — investment in private companies intended to generate 
socially desirable results in addition to financial returns (Box 1) — and the assessment of its 
effectiveness. Such investment reflects rising interest on the part of private investors about their 
contribution to societal goals?. As such, impact investments have become increasingly important both 
in responding to societal demands and in contributing to the investments needed for meeting the 
Sustainable Development Goals, when other sources of financing fall short of the massive financing 
required (United Nations IATF, 2020). 


Section 1 of this chapter briefly sets the context. Section 2 reviews some issues and challenges 
pertaining to impact investing. Section 3 discusses the measurement of impact. Section 4 introduces 
the rationale for the EIB-GDN deep dive programme. 


2 We use the word “societal” as a shortcut to encompass social and environmental dimensions. 
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1. The rapid rise of impact investing 


Achieving the Sustainable Development Goals will require massive investment. In 2014, the United 
Nations Conference on Trade and Development (UNCTAD) estimated that the total investment needs 
of emerging market and developing countries could be as high as $3.9 trillion a year, only about $1.4 
trillion of which could be financed from domestic revenue sources, official external aid, and private- 
sector financial flows and investments (UNCTAD, 2020). Given the magnitude of the gap, increases in 
official financial flows — which account for less than 5% of financing needs — must be supplemented by 
increases in developing countries’ own domestic resources and by a substantial increase in the private 
sector’s funding of and investments in the SDGs. 


The situation has become even more challenging since those estimates were made. As a result of the 
COVID-19 shock to the world economy, financial flows to emerging markets and developing countries 
may decline by as much as $700 billion in 2020 compared to the level in 2019 (OECD, 2020). The 
negative impact of the pandemic on the financial flow to developing countries has been at least 60% 
larger than the impact of the global financial crisis in 2008 to 2009. 


Box 1: The origins of impact investment 


Recognition of the role of the private sector with respect to social and environmental goals came 
gradually. A first step was the observation that production processes as well as the final good or 
service affect society, and that invested money should not support activities that society considers 
immoral or unethical. Avoiding unethical or welfare-reducing activities was the approach of the first 
socially responsible investing mutual funds and the establishment of development finance 
institution financing of the private sector in developing countries — for example, by the CDC Group 
or the International Finance Corporation (IFC) in the 1950s (O’ Donohue et al., 2010). 


The environmental movement in the 1970s and the movement for divesting from apartheid South 
Africa in the 1980s followed a similar logic (Bugg-Levine and Emerson, 2011; IFC, 2019). This 
evolution towards corporate social responsibility was concerned more with ethics and avoiding 
negative impact than with actively pursuing specific social and environmental objectives. 


Barman (2015) dates the “invention” of impact investing to about 2007, when the Rockefeller 
Foundation noted the gap between the trillions of dollars of social and environmental needs 
around the world and the resources committed by global philanthropy, which amounted to only 
about $500 million at the time. Several market-based solutions — including microfinance, 
community development and clean technology (Rockefeller Philanthropy Advisors, 2009; Monitor 
Institute, 2009; O’Donohue et al., 2010) — were being promoted at the time; however, these 
initiatives were not coordinated around an objective of societal impact, and investors did not include 
the pursuit of such impact in their objective functions. The Rockefeller Foundation brought together 
experts and mainstream investors to explore the conditions of their involvement. One conclusion 
reached was that such involvement needed to be based on evidence. From the start, the 
requirement of assessing impact was an intrinsic part of the impact investing concept. 


The importance of assessing investment activities that involve both financial and social 
dimensions is reflected in the core characteristics that define impact investing (GIIN, 2019): 
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e Intentionally contribute to social and environmental impact and earn a financial return 
e Use evidence and impact data to design the investment 
e Manage impact performance 


e Contribute to the growth of impact investing 


Recognising this need, international donors have increasingly focused on new and innovative 
financial instruments to support private-sector investments in developing countries. The rationale — 
that there are market failures that prevent efficient private investment — is twofold. First, there may 
be information failures that increase the perception of risk in SDG-related investments in developing 
countries. Second, there are missing markets for social and environmental benefits, and public money 
may be needed to orient investor choices. A key challenge is that of “additionality,” or the idea that 
donor intervention should not be a substitute for private choices that would have been made 
otherwise. In recent years, impact investing has emerged as a potential bridge between the financing 
requirements of the SDGs and official development assistance. 


However, impact investing is also under a market-driven dynamic supported by increasing demand 
for societal benefits from citizens, consumers, workers, investors, advocacy groups and 
governments. A new class of investors has emerged who jointly focus on financial profitability or 
sustainability and a positive contribution to society. Impact investing thus appears both as a market- 
led trend on international financial markets and as a channel to stimulate international donors’ efforts, 
which often reinforce each other. 


These investors cover a spectrum of organisations from family offices and foundations, to large 
banks and investment houses that are increasingly entering into the space. The term “investor” may 
include asset owners (for example, pension savers), asset allocators (for example, financial advisors) 
and asset managers (those managing investment funds and client accounts). A simple distinction can 
be made between organisations like commercial entities, which invest with impact (where the primary 
focus is on financial rate of return alongside social impact) and those investing for impact (where the 
primary focus is on achieving social impact) (West, 2019). Major players in the investing for impact 
category are development banks that specialise in providing risk capital and supporting private-sector 
projects in developing countries. They can be either national institutions that are wholly or partly 
owned by governments, known as bilateral development financel institutions (DFIs), or private-sector 
arms of multilateral development banks (MDBs) that have been established by more than one country. 
These multilateral development banks — or dedicated organisations within them, for example the 
International Finance Corporation (IFC) as part of the World Bank Group — extend loans and equity 
investments to private firms in developing countries. They do this by investing in fund managers (so- 
called “intermediated” investments) or by providing capital “directly” to companies. 


Impact investing has proven to be fairly resilient to the effects of the COVID-19 shock, particularly 
compared with foreign direct investment and portfolio investment inflows into developing countries 
(Kharas, 2020). It currently accounts for less than 10% of total net flows to developing countries. Size 
estimates vary among sources. The Organisation for Economic Co-operation and Development (OECD) 
analysed a portfolio of $70 trillion assets under management (OECD, 2020); around $17.5 trillion met 
the environmental, social and governance (ESG) criteria, while only a small fraction of these assets 
(around $444 billion) was labelled as impact investing. Using another methodology and database of 
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1 720 impact investors, the Global Impact Investing Network (GIIN) estimated the size of the impact 
investing market in 2020 at $715 billion (GIIN, 2020a). 


The successful expansion of the private impact investing market appears to be a crucial component 
of the SDGs, particularly in developing countries. However, it fuels questions about how impact 
should be assessed and draws increased attention to impact measurement within the development 
finance community and the group of impact investors (see, for example, GIIN 2019). Even among 
international development organisations that have spent decades grappling with the issue of impact, 
according to a report by the Overseas Development Institute, “there is too much ambiguity and 
confusion about what ‘impact’ is, how it should be defined, how to measure it and what kind of 
measurement is sufficient” (Hearn and Buffardi, 2016). 


2. Pursuing impact: Concepts, stakeholders and motivations 


2.1. What is impact? 


Just as investors have diverse financial return expectations, different definitions of impact shape 
diverse attitudes to seeing evidence of impact “returns.” The various ways in which investors 
conceptualise impact give “different weight to the importance of causality between an intervention 
and its effects and the measurability, breadth and timeframe of those effects” (IMP, 2020a). These 
expectations and definitions shape what investors mean by measuring impact. In practice, however, 
impact investors use the term “impact” in one of three ways?. 


First, impact refers to any type of change. Under this definition, most investments have direct and 
indirect, intended and unintended impacts on consumers and users, the environment and society at 
large (OECD, 2019). Impact is a synonym for any kind of positive or negative change, referring both to 
specific “outputs” (such as a company’s products, services and practices) as well as “outcomes” (the 
effect of those products, services and practices on people and planet). This is the most general use of 
the term impact — a catch-all that many investors have now added to their financial lexicon to signify 
that a non-financial element is under consideration; for example, the “impact of a real estate 
investment” (IMP, 2020b) or the impact of a microfinance institution on a village in India. 


Second, impact as contributions to higher-level change. In this definition, impact “refers to change 
occurring in communities or systems” (So and Staskevicius, 2015). Impact is the intended changes at 
the highest level of a theory of change or the top of a results framework. As defined by the 
International Initiative for Impact Evaluation (3ie), impact is “how an intervention alters the state of 
the world ... [so] impact evaluations typically focus on the effect of the intervention on the outcome 
for the beneficiary population” (White, 2013). This framing is often used by practitioners and project 
managers emerging from the world of international development, and wider social science. In other 
words, it describes how projects are contributing to longer-term outcomes concerning poverty, 
prosperity and well-being, for example the impact of a health fund on mortality and morbidity. 


Third, impact as causal change. In this definition, impact is the increase in the “quantity or quality of 
the enterprise’s social outcomes beyond what would otherwise have occurred” (Brest and Born, 2013). 
This can be termed the more academic definition of impact, which is at the heart of the growing field 
of impact evaluations that seek to assess the changes that can be attributed to a particular project, 


3 For more on this discussion see Hearn and Buffardi (2016). 
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programme or policy*. Such evaluations seek to predict the outcome had the investor and enterprise 
not done what they did (the “counterfactual” scenario)°. Impact is therefore a specific change that is 
caused by the investment; for example, the additional profit accruing to small enterprises as a result 
of accessing a microfinance loan. 


Many of the fundamental challenges — and confusions — about impact measurement can be traced 
to the difficulty of articulating what “impact” means and the different ways in which the term is 
defined (Thornley and Locascio, 2018). One investor may use the term in a narrow sense while others 
may use a broader definition. This can cause misunderstandings about the purpose and scope of 
impact assessments and evaluations. Even among multilateral development banks there are a range 
of uses: some see impact as a synonym for “results,” while others see it as an academic term for 
counterfactual scenarios, or for the indirect and induced effects beyond outcomes (EIB, 2019). 


To move beyond definitional debates and examine what impact looks like in practice, a collaborative 
effort by over 700 organisations under the auspices of the Impact Management Project (IMP) 
produced a set of shared fundamentals for measuring, managing and communicating impact. This 
deconstructs impact into five dimensions: understanding what outcomes are contributed to; who 
experiences those outcomes; how much those outcomes affect people or planet; what the 
contribution was to impact; and what risks exist to impact being different than expected. These IMP 
dimensions are set out against a simplified results chain (Figure 1)°. 


Figure 1: The impact value chain and the dimensions of impact 
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Source: adapted from Clark et al. (2004) and IMP (2019) 


4World Bank. 

5 IMP contribution. 

6 In the W.K. Kellogg Foundation 2004 “logic model,” outcomes refer to effects on programme participants, while impacts 
describe intended or unintended changes beyond participants in the society at large within seven to ten years as a result of 
programme activities. 
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2.2. Impact from what or whom? 


Who creates impact and through what actions? The generation of impact involves a chain of 
stakeholders (Figure 2). Funders including individual and official donors as well as asset holders provide 
financial resources either directly to investees who provide impact-producing goods and services, or 
to financial intermediaries who reach out to investees. Brest and Born (2013) distinguish three 
categories of impact generation: enterprise impact, investment impact and non-monetary impact 
(Figure 2). 


Figure 2: Stakeholders in impact investment 
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Enterprise impact, the most straightforward (once a credible counterfactual is considered), is “the 
social value of the goods, services or other benefits provided by the investee enterprise.” This is the 
category of impact that evaluations mostly consider (and this is the focus of the EIB-GDN deep dives 
discussed later). 


Investment impact is “a particular investor’s financial contribution to the social value” created by 
the enterprise in which the investment takes place (Brest and Born, 2013). There are several routes 
through which an investor may generate impact: he or she may provide additional capital, offer a lesser 
cost (through concessional financing or better terms over longer time horizons with greater flexibility), 
or present different sets of incentives. These may be organised through different financial instruments 
(debt, equity and guarantees) suggesting that financial structuring matters for impact and is not 
neutral. 


Non-monetary impact “reflects the various contributions, besides financial resources, that investors, 
fund managers and other parties may make to the enterprise’s social value” (Brest and Born, 2013). 
This can be interpreted as the additional impact unlocked through explicit means, such as the provision 
of technical assistance, and implicitly through values and demonstration effects. A 2016 study funded 
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by the Danish government identified the following types of non-monetary impact (Koening and 
Jackson, 2016): 


e Signalling — providing a stamp of approval, providing credibility, attracting other investors, 
acting as an honest broker. 


e Knowledge — strengthening the quality of the investment model and technology, sharing 
knowledge, building the capacity of local partners, facilitating technology transfer, publicly 
sharing experiences and learning (beyond project boundaries). 


e Demonstration — supporting innovative pacesetters to de-risk new business models, attracting 
capital in lower income countries and frontier markets that are not (yet) able to attract 
significant levels of commercial capital. 


e Aggregation — supporting projects at regional or global level for aggregation of opportunities, 
diversification of risk and cross-boundary sharing of experience. 


e Standards — promoting high environmental, social and governance standards in investee 
companies, financial institutions, funds and at industry level. 


Non-monetary impact can have “systemic effects” beyond the individual investee companies to 
influence wider sector and market performance. As Desai et al. (2017) note in their research, 
“individual firm activity can make markets in developing countries work better through improved 
competition, demonstration effects of a profitable business model that may not have been tried before 
in that context, or by spurring sector wide policy, legal, or regulatory reforms.” 


Figure 2 adds another category of stakeholders, “knowledge brokers,” to recognise that producers of 
impact-related knowledge also belong to the impact generation chain. Impact assessments inform 
project design, implementation and results. The knowledge they create can inform project 
implementation as well as future investments, and can lead to better impacts over time and a better 
understanding of the relationship between activity and impacts. Evaluative research should be 
considered part of impact investing through the mobilisation of measurement systems, and the use of 
research to document externalities and inform the complex network of economic, social and 
environmental impacts. This is one of the rationales behind the EIB-GDN deep dives. 


2.3. Does impact imply below-market rates of return? 


Brest and Born (2013) ask whether investors can “both make money and make a difference.” One 
argument is that without making enough money to cover costs and invest in development, any 
difference they might have is likely to be unsustainable. The issue is the kind of financial return that 
impact investing can achieve, and the compatibility between maximising shareholder value through 
financial returns (Friedman, 1970) and targeting stakeholder value. Decisions to invest for impact are 
supposed to take place in private markets where investments respond to and compete on profitability 
requirements, and private returns need to be made commensurable. This relates to a central dilemma 
in welfare economics: perfect markets lead to pareto optima but generally not to social optima (and 
imperfect markets lead to neither). Could impact investing bridge the two’? 


From aconceptual perspective, this may seem doubtful. Given the historical philanthropic inspiration, 
expectations of potential impact investors about financial outcomes may be less than for regular 


7Pareto optima deal with effectiveness: it is not possible to improve the situation of one actor without deteriorating that of 
another one. This clearly does not imply a socially desirable (not even mentioning optimum) allocation of resources. 
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investments (for example buying stocks or bonds to pursue profits). Why would adding objectives that 
are difficult to capture privately and that may incur additional costs not affect financial returns? 
Answers include benefits in terms of image, satisfaction and fidelity of the consumer base; returns in 
terms of capacity to innovate in new and promising market segments; and discovery of profitable 
investments that would have been overlooked. Visionary impact investors can then allocate resources 
to companies and push them toward impact-creating decisions®. Accordingly, “impact investing” 
would be good, competitive investing. Fundamentally, however, most environmental and social 
benefits have characteristics of public goods, however imperfect, and are therefore underproduced 
through market forces alone. Producing them should not be expected to systematically generate 
commercial returns. 


Causality between commercial and societal returns can go both ways: environmental and social 
performance may contribute to profitability, and profitable firms may be better able to engage in 
ESG or impact investing (Desai et al., 2017). Ultimately, any trade-off between financial and non- 
monetary returns is an empirical question. However, the empirical jury is still largely out (Box 2). 


If and when markets do not assign financial values to societal impacts, there must be some outside 
willingness to “pay” for them. The financial return of an impact investment can be abstractly thought 
of as an algebraic combination of a market return (for a typical investment) and the financial value 
assigned to the expected societal impact. Impact investments can be seen as synthetic vehicles that 
combine a traditional investment, producing market returns, and a commitment to societal impact, 
which has a market value that the investor or a third party may be willing to pay for. This provides an 
original model for thinking of impact investments as a virtual combination of public or private 
philanthropy — or socially oriented investors — and market-based investment decisions. This addresses 
the potential disconnect between financial returns and societal impacts by allocating them to different 
types of investor. Chowdhry et al. (2019) studied the joint financing between profit-motivated and 
socially motivated (impact) investors and argued that the latter’s financial claims counterbalance the 
former’s focus on profits. They describe a continuum of impact investments, from pure non-profit 
status for higher valued social projects, to pure commercial investment (when impact is fully 
compatible with market returns); typical impact investment thus becomes a contingent social contract 
between socially motivated and commercially motivated investors. For such a model to make sense, 
however, more knowledge is needed about societal impact and how to measure it and its value. 
Blended finance experiments have taken place in which public money (or money from philanthropic 
organisations, development finance institutions or individuals) nudges private investors, under the 
promise of a commercial return, towards producing public goods. This combination of differently 
motivated investors might create opportunities for new and promising models of public-private 
partnerships. 


8 The example of “fair trade” labels shows that companies or consumers are willing to buy their inputs and products from 
providers who have fair labour practices, environmental safeguards, and so on, and that willingness can be reflected in 
accepting a significant premium. A Swedish study, for example, claims that the premium could be up to one-third of the 
price of a good (Schollenberg, 2011). 


16 Measuring impacts 


EIB-GDN Programme - Measuring impacts 


Box 2: Is there a trade-off between financial and societal returns? 


Limitations of empirical studies include the following: uncertainty about what is covered by, and 
measured as, impact; endogeneity; selection biases; different characteristics of investment (in terms 
of sector, geography, timing, and so on); and diverging investors’ expectations and objectives 
(between development finance institutions and private impact funds, for example). The dataset and 
sample of investors may be incomplete; it may include types of investment that belong to “softer” 
categories of sustainable, responsible or ESG investments; it may be biased by the fact that only 
willing respondents take part; it may fail to differentiate between investors looking for market 
returns and those satisfied with below-market returns; and there may also be omitted variables, 
such as the quality of fund managers. 


Results on impact investment returns are mixed: some impact investors have achieved 
commercial returns, others have not. Desai et al. (2017) show that financial returns depend on the 
sector rather than geography and country income level. Rangarajan (2019) shows that public market 
investments yield different returns across different “impact” sectors. There may not be a general 
relationship between financial and social returns. Another empirical question relates to the 
sustainability of societal and financial returns. For example, the growth of societal benefits (through 
impact investing or ESG) may lead to declining societal productivity, which might translate into lower 
financial returns and remove any competitive advantage from impact investing. 


Reviewing more than 2 000 empirical studies, Friede et al. (2015) find that roughly 90% reveal a 
non-negative relationship between non-financial (as measured by ESG) and financial 
performance, and that the large majority find a positive relationship. Moreover, the positive 
relationship appears to be stable over time. On average, the IFC’s equity projects in impact investing 
have achieved financial results in line with or better than the evolution of the Morgan Stanley Capital 
International (MSCI) Emerging Market Index over 1988-2016, proving the possibility of being 
financially sustainable over long periods while investing for impact (IFC, 2019). These results suggest 
that it may be possible to invest for social impact while achieving reasonable financial returns. In 
the GIIN’s 2019 annual impact survey, Mudaliar et al. (2019) find that most investors’ financial 
performance was in line with expectations (14% over-performing, 77% performing as expected, and 
9% underperforming). Using survey data, Castellas, Findlay and Addis (2016) find that financial 
returns met respondents’ expectations (but an important caveat is that expectations are declared 
subsequently and are self-reported; investors may try to improve their image). Other studies, 
summarised by the GIIN, report average returns for impact funds seeking market rates and show 
that market rate-seeking impact investments can earn market-like returns in certain instances. 
Nofsinger and Varma (2014) find that socially responsible mutual funds outperform the market 
during crises and underperform during non-crisis periods. 


However, Desai et al. (2017) find that that there is no causal link between ESG and lower profits 
or any evidence that better ESG performance leads to higher financial performance. Using industry 
data and controlling for vintage year, fund size, sequence and geography, Barber et al. (2019) find 
that the internal rate of return of impact funds was 4.7% lower than that of all venture capital. They 


infer that investors in impact funds are willing to accept lower returns than traditional investors. 
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Kovner and Lerner (2015) find that community development venture capital funds are less likely to 


go public or to be acquired than traditional venture capital funds. A sample of social investments by 
the UK government yielded a -9.2% financial return (SIRC, 2015). 


3. Assessing the impact of impact investing: Issues, methods and practice 


3.1. How to assess the impact of impact investing 


A hallmark of impact investing is the commitment of the investor to measure and report the social 
and environmental performance of their underlying investments (GIIN, 2020b). Unlike purely 
financial investments which have outcomes that are more easily assessed relative to markets, impact 
investments have benchmarks on which there is less consensus. Monitoring and evaluating the results 
of projects has been a core business activity of development banks for decades, drawing on well- 
established methods in the fields of development finance and international development more widely. 
These activities are usually housed in an evaluation function that is independent of the line 
management. 


What final outcomes should be considered and measured in determining whether an investment 
has impact and how should they be judged? The objective behind impact investing seems to point 
towards effects that are beneficial to groups of individuals or to society. However, the notion of 
benefits is multipronged and subjective: one classically distinguishes social, economic and 
environmental benefits (see, for example, Bugg-Levine and Emerson, 2011); there may be debate on 
whether a given effect is a benefit (and for whom); there may be tensions and contradictions between 
outcomes in terms of positive value for society; and the overall effect (Bugg-Levine and Emerson’s 
“blended value”), which is the net result of a variety of partial specific effects along various dimensions, 
cannot be directly observed and must be broken down into elementary effects that can be measured 
and interpreted within given value scales. How can these elementary effects be combined to get an 
overall sense? Such aggregation implies a scale of values that may differ among individuals, groups and 
societies, and be unstable over time. This value scale is not explicit at any given time. The aggregation 
problem involves questions of summation (how to add disparate elementary effects) and of scale of 
priorities (how to consider “positive” effects along some dimensions and “negative” effects across 
others). Impact assessments may be thought of as providing packages of disparate pieces of 
information that only ongoing public debate can sort out and transform into a tentatively, and possibly 


|“ 


temporarily, shared perception of any overall “impact.” 


Shared perceptions may evolve over time, not maturing into any stable consensus. This might 
translate into difference and even incompatibility between the prior expectations of the benefits of 
the intended impact and the subsequent perception of their value, even when the intended impact 
has been achieved, because the focus and value perspectives may differ before and after. The 
aggregation challenge introduces the inevitability of a time consistency problem in impact assessment. 


Impact assessments should document specific impacts — negative or positive — and avoid providing 
aggregate synthetic judgments. Ruff and Olsen (2018) argue that impact commensuration should be 
separated from the description of what happened. Impact assessments require further interpretation 
and judgment to evaluate whether impact investing has benefited society. 
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3.2. Broad challenges 


Recent initiatives to increase the evidence base about impact often use experimental research 
techniques to isolate precise effects on incomes, livelihoods, health and well-being outcomes. Efforts 
have remained limited in scale and coordination. “Much of the industry’s reporting on impact has 
relied heavily on individual stories of successful entrepreneurs, enterprises, or employees, 
accompanied by input and output data (e.g., dollars invested, numbers of investee businesses, number 
of jobs created, etc.),” writes Jackson (2013a, p. 610). What is the benchmark for assessing impact? 
Reeder and Colantonio (2013) list nine challenges that evaluations should address’. For simplicity, they 
are reinterpreted here and grouped under three headings. 


Identifying outcomes and moving to impacts 


How does one move from outcomes to impacts? Development banks and impact-oriented private 
investors have arguably focused on tracking output-level changes such as the number of jobs created 
or microfinance loans disbursed: outputs are easier to measure directly and are often assumed to 
proxy potential positive impact creation. The links among inputs, actions and results have both prior 
and subsequent dimensions. Advance logical reasoning (also known as the “theory of change”) 
proposes a model of causal links between activities and outcomes (Hehenberger and Harling, 2018) 
using science- or evidence-based arguments that explain how activities will lead to outputs that lead 
to outcomes. A theory of change should be built in advance and encourage thinking about all potential 
(negative and positive) effects. It conceptualises an investment and inspires the choice of relevant 
indicators. 


Jackson (2013b) notes the benefits of using a theory of change to evaluate impact investing: 


e It is a cost-effective way to undertake a continuous systematic analysis. 

e It helps investors understand their impact and adjust their strategies. 

e |f public, it can help engage stakeholders and sustain a virtuous circle of feedback, adjustments 
and impact. 

e Itis compatible with other dimensions and methodologies of evaluation. 

e |t helps shape the contours of subsequent impact assessments, which may be partly inspired 
by the original theory. 


Reisman and Olazabal (2016) add that a theory of change increases the visibility of change processes 
and helps define the investment’s underlying assumptions that need to be tested. 


Determining causality 


The International Association for Impact Assessment (IAIA, 2009) defines the impact of an action as 
the difference between what would happen with it and without it. To have impact, an action must 
create additional effects over the counterfactual. This definition is conceptually clear and convincing 
but difficult to put into practice. The counterfactual is often implicitly described as the absence of 
investment, and impact is measured as the difference between investing or not investing in a project 
or company. This fails to recognise potential alternative uses of the invested money or that another 
investor might invest in the same project or company. The definition of a relevant, credible and valid 


Their work builds on insights by Barrow (1997), Coccossis and Parpairis (1992), Hughes (2002), Vanclay (2003) and 
Hehenberger et al. (2013). 
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counterfactual is crucial to impact assessment, requiring research-based approaches, and 
sophisticated approaches and tools. 


Establishing causality is central to impact assessment. It requires constant awareness that 
“correlation does not imply causality.” To establish causality, the assessor must control the biases that 
affect impact analysis, including the following: 


e Selection biases (systematic differences between treatment and control groups that explain 
some of the observed differences in results). 

e Contamination (exposure to factors other than the intervention under study). 

e Attrition (withdrawal of people included in the groups). 


Assessing value 


Discussion about the overall impact and its value for society is generally incomplete. Communication 
of impact often focuses on an investment’s positive effects, ignoring potential conflicts with other 
sectors or the fact that an investment may benefit some groups but hurt others. Focusing on positive 
effects alone is inappropriate: the objective of impact investment is to have a positive impact on 
society as a whole. Assessments of impact investments are necessary inputs into a deliberation process 
that leads society to decide, over time and after potentially conflictual debates, in which direction to 
move. 


Complete impact reporting relies on many impact-related indicators, the choice of which is an 
important part of the assessment. For example, “balanced scorecards” — a management tool 
frequently used by impact investors — assess a company’s performance along several dimensions, 
based on a variety of indicators for each. Analysis leads to the attribution of a synthetic rating for each 
dimension, which echoes the aggregation problem. The standard economic approach to facilitate 
aggregation relies on the monetisation of performance along each dimension; however, this alone 
cannot determine the choice of appropriate weights for each monetised indicator. 


Beyond the value judgments attached to the synthetic rating, using multiple indicators faces two 
additional issues. The first is the choice of indicators. Spiess-Knafl and Scheck (2018) report broad 
agreement on the specific, measurable, attainable, relevant and time-sensitive (SMART) principles. 
Concerns should be over quality (the nature of the change), quantity (the scope of the change) and 
time (by when the desired change should have taken place). The second is the potential trade-off 
between clarity and precision. A small number of aggregated indicators is desirable for clarity, whereas 
a battery is preferable for precision. Grabenwarter and Liechtenstein (2011) discuss the limitations and 
shortcomings of one commonly used indicator — CO2 emissions. 


3.3. Who does what? 


Approaches to measuring and managing impact within the impact investing market range from 
reports that describe observed changes without formally relating them to specific actions, to 
demanding scientific impact evaluations that detail causality and attribution. 


Spiess-Knafl and Scheck (2017) report that more than 150 tools and methods are used for assessing 
social impact, with no consensus to guide choice among them. They argue that four factors determine 
the choice. Reeder and Colantonio (2013) draw a typology by distinguishing between two different 
cultures in the use of instruments: system builders are more likely to use statistical tools such as 
regression analysis, whereas case-by-case supporters tend to focus more on qualitative information. 
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The two cultures have different purposes and are based on different forms of human relationships. 
According to them, most impact investors belong to the case-by-case culture. Sophisticated analytical 
techniques are too technocratic to reflect the complexity and the special nature of the project being 
assessed, and of the link between investment and results (for example, the possibility that results may 
depend on an influential fund manager and not be reproducible). 


In a landscape study carried out on behalf of the UK government, The Good Economy (2018) adopts 
a different typology of use and distinguishes between organisational-level and project-level 
approaches (which may co-exist within a single institution). At the organisational level, the general 
approach is to go beyond financial metrics and link organisational effectiveness indicators to project- 
specific results (EIB, 2019). Most investors have a results framework (for development finance 
institutions) or impact theses (for fund managers) in place which contain or are linked to a set of 
indicators reflecting desired changes occurring at the output and outcome level. Capturing these 
changes requires a degree of standardisation — of indicators, scorecards, processes and timelines — 
that allow investors to “roll up” data from individual investments to the portfolio/corporate level, 
which in turn allows them to report to the ultimate asset owners and in the public domain. Most 
multilateral development banks try to strike a balance between standardisation and flexibility (EIB, 
2019): some use highly standardised indicators, allow for highly tailored indicators, or try to blend both 
approaches to fulfil the demand for consistent and comparable reporting. The EIB uses a results 
measurement framework, based on a logical framework approach, as part of project appraisal and to 
enhance the Bank’s ability to monitor and report on the actual results achieved in its operations 
outside the European Union (EIB, 2017). 


At the project level, anticipated impacts are calculated by investors in order to make investment 
decisions, such as in investment committees or during due diligence, often using data provided by fund 
managers, investees, secondary data and assumptions, or sometimes by collecting primary data 
through site visits. Investors report their results based on data provided by their investees. There is a 
“value chain” of impact data flowing up from companies through to funds to the ultimate asset owners. 


Reeder et al. (2014) identify two distinct approaches. “Evidence followers” rely on existing evaluation 
evidence of some particular links in their evidence chain; for example, “an investor in girls’ education 
would not need to conduct their own rigorous evaluation to establish the relationship between 
education and improved economic opportunities for women” (IFC, 2019). “Evidence generators” 
actively seek data and evidence that outcomes are occurring as expected. Most investors exhibit both 
behaviours, following evidence on some links of their evidence chain and generating it on others. 


There are numerous ways in which investee businesses measure their impact. However, the 
Rockefeller Philanthropy Advisers (2020) note a “lack of transparency [...] on impact performance 
across the industry” as a key constraint to developing the field of impact measurement and 
management. There is an urgent need to improve the quality of impact data. However, resource- 
constrained companies often lack the time, expertise and budget to collect meaningful data on their 
non-financial performance. This leads to impact data of varying quality, “often either missing 
altogether, pulled together from third party research (of varying relevance), or based on poor proxies 
pulled from operational financial data such as sales figures” (IMP, 2019). Self-reported output data 
suffer from numerous biases, which range from over-inflating positive impacts to “halo effects” that 
underplay negative effects. The strength of the impact data chain would be improved by more reliance 
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on independent third-party data collection, notwithstanding biases always present in any 
measurement, notably in the selection and interpretation of indicators. 


While impact measurement is a defining feature of impact investments, “the reality is that defining 
what to measure and, subsequently, defining how to collect the appropriate data are questions that 
many impact investors struggle with” (CDC, 2019a). Half of respondents in the GIIN 2020 Annual 
Impact Investor Survey view the sophistication of impact measurement and management as a 
"significant challenge” for impacting investing over the next five years (GIIN, 2020a). Such opinions are 
reflected in investor practices in the same GIIN survey. While 91% of investor respondents measured 
what outputs occur, and 78% measured outcomes, just 32% measured “contribution to the effect 
beyond what would have happened anyway,” introducing an element of impact in the more academic 
sense. These practices shape the choice of data collection instruments and methods most commonly 
used. According to a 2017 survey by the GIIN, specific to the field of impact measurement and 
management (IMM), 60% of respondents use surveys to collect impact data, 57% use interviews, but 
only 6% have attempted more experimental methods (GIIN, 2017). 


Empirical studies (Loveridge, 2016; Edens and Lall, 2014) confirm that impact measurement practices 
hardly focus on “additionality” (in the sense of specific contribution by an investor). The IMP (2020b) 
also reports that “Having a strong focus on attribution favours rigorous [...] quantitative methods that 
[...] may limit the scope of an impact evaluation to only what can be measured under special 
conditions; are not always feasible or cost-effective given the complexity of the real world.” 


Stakeholders in the impact investing ecosystem are increasingly asking for better evidence of impact 
(EIB, 2019). Development finance institutions have been under sustained pressure to demonstrate 
results and value for money, given their use of taxpayer money (Attridge et al., 2019). This requires 
better techniques as well as data provision to demonstrate to stakeholders that investors are 
managing (and delivering) for positive impact, to refute accusations of “impact washing” and to 
generate trust (IFC, 2019). Techniques that can objectivise outcomes — and ideally estimate attribution 
— help resolve the central “paradox within impact investing”: the prioritisation of “social impact” 
without prioritising “impact evidence” (O’Flynn and Barnett, 2017). 


3.4. Methods 


Research-based impact assessments should start with a comprehensive literature review to identify 
areas and questions for further investigation and build on existing knowledge. Researchers should 
then choose a metric to describe outputs or outcomes and to consider how to access relevant data. 
Depending on time and resources, they may use secondary (existing) data or collect primary data 
themselves. Quantitative data are based on numbers whereas qualitative data are based on words or 
are non-numerical. Data collection methods can be qualitative or quantitative, depending on how they 
are implemented (for example, closed vs. open questions in surveys). Some data collection methods 
have a quantitative or qualitative “flavour.” Interviews, open question surveys and focus groups are 
more appropriate for qualitative data collection. Quantitative approaches typically rely on direct 
observation and measurement, value estimates (derived objectively through economic arguments or 
subjectively through questionnaires), experiments and surveys. 


Part of the data collection challenge in assessing societal impact is valuing societal benefits that have 
no or highly distorted market prices. This leads to different approaches depending on the degree of 
objectivity and economic foundation sought: at one extreme, rigorous cost-benefit analysis estimates 
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shadow prices to reflect real resource costs accounting for opportunity costs (under alternative uses); 
at the other, more subjective and participatory methods may be used to derive the value given to 
goods, services and societal benefits by interviewees. For example, beneficiaries may monetise the 
value of the good or service provided. Expert opinions captured with techniques such as the Delphi 
method — in which a group provides structured estimates of likely changes in outcomes (Reeder and 
Colantonio, 2013) — are useful to assess an investment’s impact. 


These quantitative and qualitative methods have different purposes and results. They require 
different time, financial and human resources. Choosing among them depends on what an impact 
assessment aims to achieve. The metric used to describe outputs or outcomes can allow before/after 
comparisons, and qualitative or quantitative comparisons of the measured impacts with a benchmark 
or with similar initiatives. The Best Available Charity Option (BACO), for example, compares the impact 
of one initiative with all options offered by other organisations (particularly in the non-profit and public 
sectors) on the same social issue (Spiess-Knafl and Scheck, 2017). The Acumen Fund, another example, 
compares the costs per unit to produce and distribute insecticide-treated nets across similar initiatives 
(Acumen Fund, 2015). 


Another dimension concerns the issue of causality (between an activity and the observed outcome 
and impact) and attribution. Not all impact assessment approaches and methods allow for or involve 
causality and attribution. Reeder and Colantonio (2013) draw a typology of assessment methods along 
two clustered dimensions (Figure 3). The vertical axis depicts the degree to which a given technique 
relies on objective vs. subjective data. The horizontal axis shows whether measurement leads to a 
battery of indicators (“multiple indicators”) or a synthetic figure (“single synthetic indicator’). The 
assessment methods on the grid are further distinguished as relating outcomes to the intervention 
being assessed and identifying the changes in outcomes associated with the intervention (the coloured 
boxes in the figure indicate whether a method adjusts and accounts for changes in outcomes). 
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Figure 3: Assessing impact: diverse methods 
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Source: Reproduced from Reeder and Colantonio (2013, p. 23). RCT, random control trials; SROI, social return on investment. 


To assess impact with attribution and causality, a range of quantitative impact evaluation methods 
as part of a well-specified theory of change can be used to set out causal pathways by which a 
programme or project influences final outcomes. Widely used quantitative methods to assess impact 
with causality include instrumental variables, regression discontinuity design, difference-in- 
differences, propensity score matching, randomised assignment and mixed methods (a combination 
of qualitative and quantitative approaches). 


Causality can be determined using natural or randomised experiments, or quasi-experimental 
techniques. Such techniques enable the control of biases, notably selection biases, through 
randomisation or the careful consideration, or estimation, of systematic differences in the samples. 
I” method. Randomised control 
trials (RCTs) have been described as the “gold standard” of experimental approaches, because they 


One lesson from academic evaluation research is that there is no “idea 


provide unbiased estimators. However, they may lack precision (Deaton and Cartwright, 2018; Deaton, 
2020) and relevance. Randomised control trials arguably rank at the top of available methods in terms 
of internal validity (that is, rigour in establishing causal relationships); however, they are less highly 
rated for external validity (that is, confidence that results can be generalised). Attempts at 
standardising impact assessment methods are likely to be self-defeating: choosing the appropriate 
evaluation method to measure the impact of a project is part of the process, and depends on data 
availability, quality and context. 


One ambitious project combining these approaches introduced the notion of a social return on 
investment (SROI), which aims to produce a quantitative (monetary) value. A consensus group”! of 
stakeholders is asked: (a) what would have happened without the company’s actions (the 


10 Akerlof (2020) describes the perverse effects of an obsession with methodological purity within the economics 
profession. 

11 Instead of voting to determine the majority opinion, a consensus group is committed to finding solutions that everyone 
can agree to: no decision is made against the will of any individual or minority. 
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counterfactual); (b) what did happen (what was added in terms of goods, services or other value for 
society); and (c) what was produced in monetary terms”. Future impact is discounted to define the 
net present value (SROI Network, 2012); the result is divided by the costs to provide a monetary ratio. 
This subjective method is arguably a less rigorous cost-benefit/effectiveness analysis. SROI informs 
causality (at least perceived causality) but may lack precision (Reeder and Colantonio, 2013); the extent 
of perceived causality may be exaggerated (Clifford et al., 2013). The approach has been criticised for 
being too complicated, because it may be difficult to account for disagreements among stakeholders 
(Arvidson et al., 2010) and the process is resource intensive”. 


There is an ongoing debate about the relationship (and compatibility) of social cost-benefit analysis 
with economic impact assessments. Recent research on environmental issues (for example, Joseph et 
al. 2020), indicates that benefits estimated through economic impact analysis using dynamic 
computable general equilibrium (DCGE) models are amenable to subsequent analysis in a social cost- 
benefit analysis framework, and combining the two approaches provides additional insights. The 
authors recommend that the environmental assessment process is reformed to include cost-benefit 
analysis complemented by economic impact analysis to provide decision-makers with more complete 
information. The relationship between social cost-benefit analysis and DCGE might have a broader 
relevance. Many approaches using rigorous impact evaluation techniques give insufficient attention to 
costs and cost effectiveness. This is important for guiding investment choices and identifying 
sustainable ways to produce impact. There is a good case for using a social cost-benefit analysis 
framework for evaluating impact investment. 


3.5. Common principles 


Impact investment funds may have an incentive to use their own approaches and metrics if the 
market is not standardised. By doing so, they will not contribute to any dynamic of standardisation. 
The lack of convergence on a method, framework or metric for environmental and social benefits is 
one reason why a market for impact investing has yet to reach maturity. 


The conceptual case for convergence is questionable. Convergence may bring clarity. Many authors 
criticise the confusion that reigns in the impact assessment market. Convergence may also bring 
transparency and credibility, because it provides a framework and ensures conformity with a set of 
rules that, however imperfect, impose a reference of quality. Convergence also provides some 
comparability (of results), which is important for evaluating and ranking investments. 


There is tension between standardisation and flexibility - and between standardisation and 
effectiveness. Big investors are reassured by standards, whereas social organisations may prefer 
assistance in developing their own methodologies (Reeder and Colantonio, 2013; Ógáin, Lumley and 
Pritchard, 2012). Standardisation has drawbacks. Fornaziere (2012) argues that potential negative 
consequences of standardisation include the distortion of capital flows towards easily measured 
sectors; the distortion of social companies towards easy-to-serve populations to increase funding; and 
an increase in pressure on social businesses to comply. 


12 The seven principles of SROI are as follows: “1. Involve stakeholders; 2. Understand what changes (for those 
stakeholders); 3. Value what matters (the “monetization principle ...); 4. Include only what is material; 5. Do not over-claim; 
6. Be transparent; 7. Verify the result” (SROI Network, 2012). 

13 Reeder and Colantonio (2013, p. 28) note that “methods such as ‘SROI-Lite’ have been developed. Sponsored by Santa 
Clara University, SROI-Lite asks enterprise managers to define the most important output they create, its unit cost, and the 
ratio of cost to successful output.” 
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Ruff and Olsen (2018) argue that there is no need for standard metrics to ensure comparability as 
they make things more rigid and complicated, and the absence of flexibility prevents comparability. 
They propose a “bounded flexibility” concept whereby for each issue area, companies can choose from 
a limited number of possible measures, enabling partial comparability. Full comparability can then be 
ensured by measuring success-to-target rates (a measurement tool also open to critique). They call for 
more professionalisation of impact assessments (of impact investing). 


3.6. Measurement as management 


Over the past decade, the evaluation conversation has shifted from debates about how to best 
measure impact to satisfy asset owners — an upwards accountability objective — to how to measure 
consistently to compare and benchmark across investments. This has included standardising 
elements of the results measurement infrastructure, such as developing ratings and assessment 
systems and catalogues of indicators like IRIS+ (a catalogue of metrics curated by the GIIN that 
investors can use to track results across portfolios and self-report on their achievements)**. The 
development finance institution community has made efforts to harmonise results indicators through 
the Harmonized Indicators for Private Sector Operations (HISPO) initiative, a collaboration of 25 
development banks to set out 27 reporting indicators for international financial institutions’ shared 
clients, drawn from over 400 metrics that were reviewed? 15, 


Typically, investors have focused impact activities on anticipating potential social and environmental 
impacts, risks and financial returns when deciding which companies they should provide capital to 
(Loveridge, 2016). This reflects the fact that the most important decision — and potentially the greatest 
determinant of the depth and breadth of positive impact creation — can be which companies to invest 
in. After the investment decision, outputs are tracked as a proxy for outcome achievement. 
Contribution, if factored in, is mainly considered before an investment is made, including through the 
requirement of “additionality” as articulated by the multilateral development banks to ensure their 
activities add to what is available in the market, and do not “crowd out” the private sector (IFC, 2018). 
In practice, additionality has proved difficult to measure, both conceptually and operationally (Kenny 
and Moss, 2020). 


Recently, the measurement conversation has shifted from a “rear-view mirror” approach to proving 
impact to improving and supporting future impact. This is based on the realisation that expected 
impacts are often not attained, particularly as investee business models can change — especially for 
early-stage enterprises backed by venture funds — and there are uncertainties in stimulating positive 
social change. Various initiatives have promoted operating principles that partly or fully promote this 
more integrated approach between measurement and management”. 


14 See iris.thegiin.org. 

15 See indicators.ifipartnership.org. 

16 The following initiatives have established and promoted common operating principles (alongside the IMP): the UN 
Principles for Responsible Investing (PRI), which focus on ESG risk management (UN PRI, 2013); the Principles for Positive 
Impact Finance of the United Nations Environment Programme (UNEP), which pushes investors to adopt an impact-based 
approach; the Global Impact Investing Rating System (GIIRS), developed by B Lab, a non-profit organisation that administers 
the B Corps’ fee-based certification of impact business models and performance; and the Operating Principles for Impact 
Management (OPIM) launched by the World Bank Group in April 2019, developed by the IFC jointly with asset and fund 
managers to promote transparency and discipline in the impact investing market as well as clarity about which investments 
can be called impact investment (IFC, 2019). 

17 In 2017, for example, the IFC developed its own proprietary framework (IFC, 2019), called the “anticipated impact 
measurement and monitoring system” (AIMM). 
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The term “impact measurement” is now generally used together with “impact management.” This shift 
represents an increasing demand to integrate social and environmental performance into operational 
and strategic decisions. Impact management is framed through the lens of continuous improvement: 
from New Philanthropy Capital “using your data to improve what you are doing” (Rotheroe, 2017) to 
Social Value UK “creating a culture [of impact] within an organisation” (Carpenter, 2020). There is no 
formal definition of impact management, but it is best seen as the practice of integrating impact at 
each stage of the investment and business decision-making process. 


Impact measurement and management itself is defined as the process of selecting and embedding 
social and environmental performance considerations into the investment cycle, collecting data, and 
using the information to drive decision-making (GIIN, 2020c). An impact measurement and 
management system is a set of activities that broadly covers the following: 


=" Selecting goals and indicators that are mission-aligned. 
= Setting targets and strategies most likely to achieve and reflect these goals. 


=" Measuring and analysing metrics to understand what is happening in reality. This can lead 
to full “monetisation” — attributing a monetary value to achievements across various 
dimensions — which the IFC (2019) considers as a “gold standard” among the integrated 
impact assessment frameworks because it enables credible internal and external 
comparability. 


These are commonly expanded on and turned into a measurement “cycle” with four key steps: plan, 
do, assess and review (Figure 4). 


Figure 4: The Plan-Do-Assess-Review cycle 
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To be effective as a driver of continuous improvement, impact considerations need to be embedded 
within an impact investor’s strategy, structuring, supervision and, where relevant, within 
investment exits (IFC, 2019). Impact management is a discipline much like financial management: 
policies, processes and procedures are necessary to ensure active management of an investment’s 
performance. 
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Figure 4 shows how impact management is integrated throughout the investment process. 
Measurement takes place at two distinct stages: pre-investment (advance measurement), and post- 
investment (subsequent measurement). These are synonymous with the “plan-do” steps, where 
impact intentions are articulated, and the “assess-review” steps, where actual impact is tracked. 


Figure 5: Impact measurement management across a typical investment cycle from deal screening to exit 
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According to the GIIN’s latest impact investor survey (GIIN, 2020a), a majority of the 278 respondents 
pay significant consideration to impact before investing — during due diligence (81%) and screening 
(77%). However, this drops to 62% during portfolio management. Only 36% collected impact data 
during exit, and 21% post-exit. Impact data are given more weight in pre-investment decision-making 
(articulating intended impacts) than in post-investment performance (measuring actual impact). 


3.7. Who pays for evaluation? 


Rigorous impact assessments are expensive because they require time, accurate data and expertise 
(Ruff and Olsen, 2018; Glanzel and Scheuerle, 2016). It is unclear who should pay (investors, investees, 
funders or a third party) (Harji and Jackson, 2018). There is a potential conflict of interest when a 
stakeholder of the impact value chain pays to assess the impact of their investment or production. The 
relevance of this conflict of interest varies depending on the objective of the assessment: it is clear for 
proving accountability; leads to questionable credibility for reporting impact. It is less problematic for 
organising the learning function within the organisation. Beyond the issue of conflict of interest, 
stakeholders may differ in their abilities to bear the costs, perceptions of the usefulness of impact 
assessments, internal capacity to conduct rigorous assessments and willingness to acquire them. 
Depending on the process, the learning gain may only profit investors, so investees may be reluctant 
to undertake an evaluation. Who pays has a bearing on the nature of the impact assessment, and 
determines its motives and incentives (for example, scientific rigour, communication and 
accountability) and its credibility. 


It is unclear what the potential trade-offs among feasibility, cost, usefulness and robustness are. The 
ability to bear the costs of a comprehensive scientific evaluation, with or without subsidies from 
foundations or development agencies, influences the design of the evaluative effort. Spiess-Knafl and 
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Scheck (2017) argue that the importance of the evaluation (time spent and degree of accuracy sought) 
is likely to be proportionate to the organisation’s size and suggest that companies should acquire 
internal expertise on impact assessments. In contrast, Clifford et al. (2013) note that such capability 
can be independent of an organisation’s size and complexity. Reeder and Colantonio (2013) consider 
that “gold standard levels of robustness are impractical for [the] vast majority of enterprises.” 


Scarce information is available on the money or resources investors allocate to impact 
measurement. A study by ANDE — a network of organisations supporting small and growing businesses 
— found that investors’ median spending on measurement (calculating staff time and consultant fees) 
as a percentage of the operating budget is 2.2% and ranges from 0% to 25%. The 50% smaller 
organisations in their sample spent a median of 8.1% of their budget on measurement (Edens and Lall, 
2014). No figures for impact measurement resources as a percentage of assets under management 
have been disclosed. 


Covering the costs of rigorous assessments remains an issue and is one dimension of the “research 
gap” that underlies the EIB-GDN deep dive initiative. 


4. Deep dives as a “bridge” 


4.1. The research gap 


It is useful to consider the motives and demand for impact assessments. Part of the demand is 
institutional. Evaluations play a broad governance role. For public investments, they have become part 
of due process; policy innovations and development projects require evaluation (both before and 
after). Private-sector evaluations (of impact investments) have been recognised as a condition for the 
structuring and expansion of the impact investment market. Regardless of their institutional nature, 
and whether they take place before (through evidence-based assessments relating potential 
investments to impacts through a theory of change, simulations of impact, and so on), during or after 
implementation, evaluations use and produce knowledge. This production of knowledge is useful for 
three reasons that in turn characterise three different functions of evaluations: 


1. The accountability function: evaluations are needed to demonstrate that funds are used in 
conformity with the original intention or mandate, based on the prescribed rules of 
engagement, and are efficiently and effectively directed towards the intended purposes. 
Evaluations may also demonstrate to the general public and policymakers that resources are 
properly used, that impact investment works, and that it should be allowed and encouraged 
to expand. 

2. The learning function: evaluations aim to improve decision-making (and future impact) 
through learning how project design and implementation relate to impacts (a learning-by- 
doing function). They improve the likelihood that a given idea and objective of impact may 
translate into such results. Moreover, evaluations improve understanding and documentation 
of the links between specific actions and impacts (a knowledge creation function). Thus, they 
also fulfil a global enlightenment function that extends the learning function to society at large. 

3. The promotion function: evaluations provide communication material that can promote a set 
of actions or the image and reputation of investors, intermediaries or companies. 


These various functions of evaluations may require different products and approaches. 
Accountability requires independent, credible, unbiased evaluations to be undertaken from the 
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perspective of the individuals or groups to which the various actors need to be held accountable. 
Learning requires the involvement of the actors under evaluation, who will not learn as much if they 
are not associated with the evaluation exercise. Independence may matter less for the learning 
function and may even be counterproductive. Knowledge creation requires open, unbiased minds; a 
critical approach to assumptions and results; a long-term perspective; and the ability to apply 
knowledge and methods from the literature. Promotion requires specific qualities including salience 
and the ability to tell powerful and easily transmissible stories. 


Different actors expect different benefits from impact assessments. We infer that the evaluations 
that impact investors undertake are mainly driven by the accountability requirements and the 
promotion motive. Learning is a recognised goal of evaluations by institutions, although it is complex 
to achieve because it implies a transformational use of acquired knowledge to change methods, 
practices and organisation. Chapter 3 discusses how the EIB-GDN served a learning function for 
investees. The wider part of the learning function — enlightenment for society as a whole — relates to 
the gains for the overall community beyond the payer or the institution under evaluation, and often 
comes as a by-product rather than a driver of impact assessments. However, it may be crucial for the 
development and effectiveness of impact investing, because its expansion and social role depend on 
sound, rigorous knowledge about impact and how it is created. 


Rigorous impact assessments, notably focusing on causality, can serve all these functions. However, 
while desirable from a credibility and knowledge perspective, they are expensive and time-consuming, 
and not perceived in the private sector as essential for accountability or promotion motives. The field 
of development evaluation has broadened over the last decade to include more impact evaluations, 
using academic research techniques to learn “what works,” including in the field of private-sector 
development (McKenzie, 2010). However, a typical impact evaluation by the World Bank or Inter- 
American Development Bank costs several hundred thousand US dollars, with some being as much as 
$1 million. They can also prompt subsequent assessments (“what worked or did not work” vs. “what 
is working or is not working”) but these come too late to result in real-time learning. 


Private investors have little incentive to call for full academic impact evaluations, because they do 
not correspond to their immediate needs. The challenge therefore is to find ways to encourage more 
academic research in the evaluation process. This is what we call the “research gap.” We believe that 
there would be substantial positive returns to filling the gap. Independent research-based insights and 
evaluations using sound methods provide tractability and credibility. For example, evaluative research 
has improved the design, implementation and effectiveness of conditional cash transfer programmes 
since the early 1990s. Similarly, the US government’s social experiments on housing assistance and 
negative income tax had major effects on social programmes. Using existing knowledge and methods, 
academic research introduces the possibility of rigorous, fact-based deliberation at all stages of the 
decision chain and related feedback loop, in the design and implementation of projects, and what can 
be learnt from their effects. This dynamic learning process is more important than the promotion of 
“results,” which cannot encompass the entirety of impact and are by necessity incomplete and 
potentially self-serving. Introducing academic research into evaluations brings rigour to impact 
assessments, providing a systematic, methodical approach to deal with biases, to interpret results with 
precision, soundness and credibility, and to build global knowledge. 
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4.2. EIB-GDN deep dives 


The EIB-GDN deep dives aim to address this gap — generating new research-based knowledge of how 
to conduct impact assessments that can better meet the evidence and learning needs of the impact 
investing community — and were shaped as a proof of concept. Deep dives can be defined as a 
dedicated evaluative research exercise to collect and analyse data on the outcomes resulting from an 
investment, as well as any causal impacts. They focus on individual projects — not the performance of 
whole portfolios — and can be used to complement existing approaches to organisation-wide 
evaluation and result reporting. 


Deep dives occupy the middle ground between fully fledged, rigorous impact evaluations (that are 
often not feasible in an investment context) and inferences from self-reported monitoring data. 
Several institutions, including the EIB, have begun to test different approaches to carrying out impact 
studies, partnering with academia and specialised organisations*®. Among multilateral development 
banks, for example, the Inter-American Development Bank has closely integrated impact studies into 
its project cycle. The IFC has carried out a number of “rapid evaluations” (Box 3). Other investors, for 
example the agribusiness lender Root Capital, use impact studies on a representative sample of their 
portfolio to validate their theory of change and inform course corrections, taking a mixed-methods 
and “client-centric” approach (McCreless, 2015). 


Box 3: A rapid evaluation approach in the IFC 


In 2011, only 3% of the female population of Pakistan had access to a bank account, and women- 
owned businesses faced a credit gap of $179 million. In 2015, the IFC made an equity investment 
and senior loan to Habib Bank Limited (HBL), the largest commercial bank in Pakistan. This was the 
first IFC investment made with the explicit intent to support finance for women-owned businesses. 
HBL has subsequently launched a sub-brand, HBL Nisa, to target women. HBL has also set up a 
women’s business unit, increased key performance indicators for women’s deposits, appointed a 
diversity manager, and trained staff in gender intelligence. 


The IFC used quasi-experimental methods to conduct a rapid evaluation of the cause and effect 
of the gender intelligence programme. The evaluation examined the differences between 
employees who had undergone training compared to untrained employees. Over 13 000 HBL 
employees were surveyed, with branch-level data analysed for the years 2014 to 2016. This study 
showed that branches whose managers have been trained in gender intelligence demonstrated a 
10% increase in the volume of deposits from women-owned accounts when compared to branches 
with untrained branch managers. 


Adapted from: Creating Impact, the Promise of Impact Investing (2019) 


The closest comparison to the EIB-GDB programme, and also a source of information, is the series of 
deep dives run between 2016 and 2018 by the Impact Programme, a UK government-funded 
initiative seeking to build the market for impact investing in sub-Saharan Africa and South Asia. 


18 The EIB has concluded a study on the impact on central and eastern European small and medium-sized enterprises over a 
comparator group, based on EIB and ORBIS data. Efforts are ongoing to better coordinate and pool such activities among the 
multilateral development banks. 
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These studies focused on a selection of high-impact investments made by the CDC Group, the United 
Kingdom’s development finance institution (Box 4). The Impact Programme deep dives focused on 
generating rapid insights into impact that had value for investors and investee companies alike — rather 
than comparing outcomes between a “treatment” and a “control” group, they prioritised “lean” 
methods that provided evidence of outcome and were useful for company decision-making. The deep 
dive approach has since been integrated into CDC’s Rapid Insights Toolbox (CDC, 2019b). 


Box 4: Impact Programme deep dives 


The CDC Impact Fund invests in funds and other intermediated vehicles that deliver high 
development impact, particularly to underserved populations. In mid-2016, as part of the UK aid- 
funded Impact Programme, the fund began piloting a series of insight studies. These were called 
“deep dives” and aimed to complement existing data reporting and improve understanding of how 
people engaging with portfolio companies were experiencing change. Each deep dive cycle took two 
to three months. The aim was to deploy a measurement approach that was light-touch, in-depth 
and could generate value for investors, investees and end beneficiaries. In partnership with 60 
Decibels, the studies used lean data principles, with each deep dive aiming to be: 


= Bottom-up — helping companies listen to customers, communities, employees and suppliers 
to provide actionable insight on their needs and interests. 

= Useful -— yielding data of sufficient quality to support decision-making. 

= Iterative — allowing learning, adaptation and replication. 

= Light-touch — using low-cost tools and technologies that require a minimal investment of 
time and money. 

=" Dynamic — enabling rapid data collection within a fast-changing environment. 


In the first year, deep dives into five portfolio companies took place in Ghana, Nigeria and Kenya, 
covering the agriculture, energy and information and communications technology sectors. The 


table below sets out the range of topics covered against the IMP’s five dimensions of impact. 


Who experiences change, and A deep dive into an online news platform to understand 
how underserved are they? the socioeconomic and demographic profile of readers 
What outcomes does the A deep dive into a low-cost urban Wi-Fi provider to 
change relate to, and how understand the importance of in-home internet 
important are they to the connectivity 

people (or planet) experiencing 

it? 


How much of the change occurs A deep dive into a solar irrigation company to 
in the time period in terms of understand time and cost savings from moving away 
depth, scale and duration? from manual/diesel pumps 
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Contribution. How does the A deep dive into an animal feed mill to understand how 
change compare and contribute the products create superior value for customers 

to what is likely to occur compared to the alternatives used by non-customers 
anyway? 

Risk. How likely is the outcome A deep dive into a company’s shea nut suppliers to 


to differ from what is expected? understand future selling intentions and workplace 
safety concerns 


EIB-GDN deep dives push the boundary of impact studies further to address questions about 
causality. Deep dives expand the range of methods and approaches used to assess the impact of 
investments. A more academic approach can yield higher quality impact data by including methods 
that unpack the causal relationship between observed changes and a specific intervention or 
investment. 


The underlying premise is that the potential to use techniques drawn from more rigorous research 
methods as part of impact measurement systems has not been fully explored, and “the vast amount 
of [impact evaluation] literature and experience that exists on government or grant-funded 
programmes has [so far] not been properly adapted to the realities of private sector-focused impact 
measurement, neither from a resource nor process perspective” (CDC, 2019a). However, improving 
the evidence base is not simply a matter of duplicating research methods and designs in the context 
of impact investing, as “measuring the impact of investments in a private-sector context is different 
from independent impact evaluations of government- or grant-funded programmes [...] due to 
commercial, legal or logistical reasons” (CDC, 2019a). 


EIB-GDN deep dives aim to generate knowledge on how to use impact research techniques that are 
rigorous but the “right size” for a private-sector context. The core challenge was to establish methods, 
techniques and processes to assess attribution with a “reasonable standard of robustness” (Reeder et 
al., 2014). Box 5 presents the main objectives and aspirations of the programme. Chapters 2 and 3 
outline the lessons learnt on this journey and document the extent to which the initial goals were 
validated. 


Box 5: EIB-GDN deep dives 


The pilot programme initiated in 2016 seeks deeper insights into the profile of end beneficiaries 
and how they are affected by EIB operations, using primary data, in-depth analysis and 
academically rigorous methodologies. This box summarises the objectives and goals of the 
programme, which are detailed in Chapter 2. Some of these studies also address the question of 
causality — the extent to which any observed change in outcomes can be attributed to an operation 
under the EIB’s Impact Financing Envelope for Africa, the Caribbean and the Pacific, rather than to 
any other social or economic changes — via impact evaluation methods. The programme is using and 
enhancing capacity for impact studies by training a set of talented young researchers from African, 
Caribbean and Pacific countries. 
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The three overarching objectives of the programme are: 


1. To give the EIB deeper insights into the results and impact of its investments, which it can 
share with its stakeholders, for example the committee members (representing the EU 
Member States) responsible for the African, Caribbean and Pacific Investment Facility. The 
funding for the projects under study and the EIB-GDN programme comes from this facility. 


2. To bring added value through the research for EIB client companies (direct and indirect 
clients, in the case of intermediated operations such as investments in funds). 


3. To build capacity for local researchers and foster a local research community. The GDN 
recruited talented early career researchers from Africa, the Caribbean and the Pacific for 
the programme, who received support from five global expert advisors. 


To achieve these objectives, the EIB assigned four aspirations for the deep dives: 
Aspiration 1: Answer questions that matter to EIB clients and other stakeholders. 


The companies should ultimately drive the scope and focus of the deep dives: asking questions 
that they need to answer but have not been able to tackle using their own resources. This will be 
a two-way discussion: the researchers will work with the companies to identify which questions can 
feasibly be addressed. The findings should be relevant to other EIB stakeholders, particularly the 
committee responsible for the Investment Facility for Africa, the Caribbean and the Pacific, which 
funds the projects being studied and the deep dive programme. 


Aspiration 2: Generate customer insights with commercial value. 


The core of the value proposition is to improve understanding of customers/clients (and potential 
customers in some cases): who they are, what they want, how they use the investee’s products 
and what motivates them. The specifics will vary by company, but better insights into customers 
should translate into a better understanding of how to strengthen a company’s business model, 
particularly when the findings do not fit the company’s pre-held assumptions. Insights into 
customers also translate into a better understanding of the company’s social impact, which can be 
employed for marketing purposes with insights into the value added. This should be of interest for 
companies with an explicit social/environmental mission, and/or those aiming to attract resources 
from international financial institutions/multilateral development banks/donors. 


Aspiration 3: Use methods that are rigorous, timely and right-sized. 


Once the questions are defined, the researchers will work to identify appropriate methods, 
drawing on their academic training and the support of the world-renowned experts mentoring 
them. The methods should be chosen bearing in mind the timescale of the programme and the data 
needs of the companies. The depth of the engagement should be tailored to the size and stage of 
the firm. The rigorous methodologies ensure that the evidence informing the firm’s decisions is 
accurate, relevant and meaningful. The core aim of using cutting-edge techniques is to avoid basing 
decisions on misleading findings. The researchers will need to adapt these techniques to be 
appropriate for the questions, hence the studies will differ from academic work. 
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Aspiration 4: Contribute to a new way of thinking about impact measurement. 


The techniques used in the deep dives will generally not be new. However, it remains an innovative 
way of thinking about impact measurement — putting commercial value added via customer insights 
at the core, rather than starting with questions set by academics or donors. This can bring significant 
value to companies and is the only way to proceed — deep dives can succeed only if the company is 
fully engaged, and this will be achieved only if there is commercial benefit. 


4.3. Concluding remarks 


While remaining small relative to global volumes, the impact investing segment of financial markets 
is growing fast, focusing attention on measurement and assessment of results, including social returns. 
Three main messages emerge from this chapter. 


First, impact assessments mean different things to different stakeholders. There is a gap between 
the private value of impact assessments (driven mainly by the desire to document and report success, 
and the requirements of accountability) and their social value (based on the extent to which they 
increase the conceptual and operational understanding of the economic, social and environmental 
nexuses). Bridging the gap requires analytical efforts, and external incentives and resources. 


Second, rigorous academic evaluative research benefits all stages of the investment process 
(identification, conception, implementation, and assessment of results and impact). Substantive 
inputs cannot be reduced to measurement frameworks; evaluations should continue to consider 
several approaches and models and evaluators should cultivate strong connections with academia 
(Harji and Jackson, 2018; Ruff and Olsen, 2018). The EIB-GDN deep dive programme shows how 
rigorous research can provide insights into impact creation. 


Third, the cost of increasing cooperation with academia for using more research-based methods to 
improve the quality of impact assessments is unlikely to be covered by private investors. This cost, 
which represents the public-goods dimension of impact assessments, is related to the “enlightenment” 
and “learning” functions of evaluations. Supporting the market for impact investing cannot rely only 
on investors’ engagement and generosity. Research-based evaluations applying rigour and control of 
biases should be promoted, and the focus should be more on knowledge creation than on the 
promotion and reporting functions of evaluations, while considering private stakeholders’ interests 
and motivations. It is hoped that the lessons learnt from EIB-GDN deep dives will lead to a new 
generation of research-based evaluations, thus filling the gap. 
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Summary and key messages 


The previous chapter introduced deep dive impact studies as a way to complement other evaluation 
efforts and combine methodological rigour with the time requirements of stakeholders in an impact 
investment. This chapter discusses the experience and lessons from a recent pilot programme, 
developed and funded by the European Investment Bank (EIB) in collaboration with the Global 
Development Network (GDN), which conducted 16 deep dives of investments in ten African, Caribbean 
and Pacific (ACP) states. It details the design and principal components of the deep dives, how they 
were implemented, their main outputs/outcomes (and the value added for stakeholders), and the 
lessons learnt about design, implementation, utility and use. 


The deep dives were conducted by 30 research fellows from ACP states. The researchers were well- 
trained but relatively junior, as an important subsidiary objective was to build evaluative and research 
capacity. Expert advisors and EIB and GDN staff supported the researchers in completing studies that 
passed academic review in a limited time period, adding to general development knowledge, with 
some expected to be published in peer-reviewed journals. The deep dive exercise provided lessons on 
how research can be relevant for operational and policy concerns (Chapter 3). 


Lessons have been learnt about conducting these programmes. First, deep dive research must take 
into account a lack of existing data. While trying to make the best use of data, researchers needed to 
plan and budget for primary data collection. Second, engagement with the company implementing the 
investment is key. Company buy-in is essential for research and to ensure successful cooperation and 
data sharing. Third, researchers must be flexible about research approach and methodology. Time and 
resource constraints, and the nature of the project being assessed, meant that randomised control 
trials planned in advance were not generally feasible. Other techniques to address attribution were 
needed. Fourth, conclusions need to be tailored to the limitations of the analysis. It is a challenge to 
draw firm conclusions that are not generic, that add value to the global community and that are 
operationally relevant. Fifth, reporting in the right presentational form is crucial; expert advisors, GDN 
and EIB staff and consultants supported the researchers in their communications. Sixth, research 
capacity can be built, but it is a slow process; technical capacity building was limited by the short time 
period available. Seventh, the costs of the deep dive, although significant, were not prohibitive and 
judged by the reactions of all stakeholders were outweighed by the benefits. 


Increasingly, investors, financial intermediaries and private suppliers/providers need to consider, in 
addition to their financial returns, the impact (positive or negative) of their projects on people, the 
environment and governance of their economies. Most multilateral development banks?, bilateral 
donors and the foundations that support them have established practices to understand the 
development impact of their investment programmes, including private-sector investments. Almost 
all have frameworks to track some of the direct “results” of investments, often measuring these 
impacts during implementation as well as after project completion. Most carry out evaluations of all 
or selected investments (some institutions evaluate each operation, while others take mainly a 
thematic approach, with detailed case studies of a sample of operations). 


2 See, for example, the Evaluation Cooperation Group (ECG)’s Good Practice Standards for the Evaluation of Private Sector 
Investment Operations. The ECG consists of the independent evaluation units of the major international financial 
institutions. 
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However, as noted in Chapter 1, results measurement frameworks generally only capture the direct 
impacts of projects and focus on indicators that can be aggregated over different operations. 
Evaluations can study particular investments or themes in more detail, but are generally only 
implemented after project completion, and look at impact as one of several criteria. Stakeholders often 
want a deeper understanding of the impact of particular investments, including during the 
implementation period. Furthermore, results measurement frameworks do not generally address the 
issue of causal attribution, which would need to be tackled project by project, even though some 
multilateral development banks and bilateral donors, such as the World Bank, the Inter-American 
Development Bank and the Millennium Challenge Corporation, undertake numerous rigorous impact 
evaluations. Few institutions have made extensive use of impact evaluations that meet the highest 
academic standards. This is partly because of cost considerations and partly because traditional impact 
evaluation studies yield results after the investment project has ended. Assessing the evolution of 
impact during implementation, rather than only after the investment has ended, can help institutions 
to adapt practices and strategies to maximise impact. 


The previous chapter introduced deep dive assessments as a way to complement other evaluation 
efforts and combine methodological rigour with the time requirements of stakeholders in an impact 
investment. What has been the experience with EIB-GDN deep dives and what lessons can be drawn 
from them? Deep dives are relatively new in the evaluation field and vary in scope and character 
(Chapter 1). This chapter discusses the experience and lessons from a recently completed pilot 
programme, funded by the EIB and implemented by the GDN. It details the design and principal 
components of these deep dives, how they were implemented, their main outputs/outcomes (and the 
value added for stakeholders) and the lessons learnt in their design, implementation, utility and use. 
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1. What went into the EIB-GDN deep dive programme? 


The EIB-GDN project was started in 2017 and ran for three years (organised in three one-year cycles). 
It was conceived through a series of discussions starting in 2014 between the GDN and the EIB. The 
EIB was interested in complementing its impact measurement framework by developing the research 
component of its operations and involving developing country researchers. It identified the GDN as a 
partner to build research capacity in developing countries, and initiated a partnership focused on 
understanding the development impact of the Impact Financing Envelope (IFE) for Africa, the 
Caribbean and the Pacific regions. 


The project aimed to strengthen the capacity of young researchers in developing countries while 
producing research-based insights into the development impact of investments funded by the EIB 
under the Impact Financing Envelope. Specifically, the project designers wanted to: 


e Answer questions relevant to EIB clients and other stakeholders. 

e Be rigorous in methods. 

e Contribute to a new way of thinking about impact measurement that combines the needs 
of academic research and investor interests. 


The programme aimed to match projects (in selected countries of Africa and the Caribbean) with 
researchers based in, or with a background in, those countries. The aim was to bring contextual 
knowledge and legitimacy by mobilising research fellows from beneficiary countries. This provided a 
unique opportunity for the researchers to acquire insights into projects financed by one of the biggest 
multilateral development banks in the world, while strengthening the pool of local experts. 


To ensure that the studies met high standards of technical rigour, the GDN matched the researchers 
with a team of expert advisors. Chosen by the GDN for their experience, competence and high profile 
in the field, they ensured research met appropriate academic standards. This was particularly 
important given the innovative nature of the programme and the need to produce quality work in the 
fast-paced environment of the private sector in developing countries. Meeting quality objectives under 
tight time constraints and looking at complex investments requires skills, expertise and experience. 


EIB staff were heavily involved in supporting the programme. The EIB had been expected to play a 
role in identifying the operations for analysis, making introductions to the relevant EIB clients, and 
ensuring strategic alignment between the studies and EIB objectives. In reality, the involvement of the 
EIB was more extensive (Figure 1 illustrates the key role of the EIB and the GDN among the 
stakeholders). Because the research fellows were relatively inexperienced, particularly in a private- 
sector context, the EIB team played an important role in explaining the nature of the investments and 
the underlying business models. They also acted as facilitators between the researchers and the firms, 
provided technical reviews of documents and edited reports, drawing on the technical expertise and 
subject knowledge of staff in the Bank’s Economics Department and the Independent Evaluation 
Division. 
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Figure 1: Deep dive stakeholders 


Deep Dive Relationships (shaded arrows indicate financial support; 
others indicate non-financial interactions, such as advice, consultation) 


Financial Production or 
EIB m Intermediary = Service Enterprise 


(Company) (Firm) 


Research 


Fellows 


External 
Advisers 


NB: This is an application of Chapter 1, Figure 2 to the case of deep dives. 


The research outputs of the programme were called “deep dives.” They were expected to answer 
questions relevant to EIB clients and other stakeholders; to be rigorous in methods; and to contribute 
to a new way of thinking about impact measurement that combines the needs of academic research 
and investor interests. As shown in Figure 1, the deep dives were written by researchers, and benefited 
from interactions from various stakeholders. They provided in-depth insights into specific research 
questions, shaped through interactions between the researchers, expert advisors, EIB and GDN staff 
and the companies that benefited from EIB funding. They were developed under the practical 
constraints of data availability and a timescale of one year per study. The following subsections detail 
how the EIB-GDN deep dive programme was implemented. 


1.1. Recruitment of research fellows 


The first task for each cycle was to recruit the research fellows through a competitive application 
process in three successive cycles. The process was geared geographically to facilitate matching 
between the selected researchers and a preliminary list of projects to be evaluated, proposed by the 
EIB, attempting to target francophone as well as anglophone researchers to make it easier to carry out 
studies across most countries of EIB operations in the relevant regions. The call focused on candidates 
from low- and middle-income countries in Africa, the Caribbean and the Pacific, since the source of 
funding was related to the Cotonou Agreement, which covers those regions?. Given the location of the 
projects selected for deep dives, most of the research fellows (except one from Haiti in the Caribbean) 
were from countries in sub-Saharan Africa, and lived and worked in the region (Table 1). A total of 30 
researchers were recruited during the programme (as listed in the “Biographies” section of the book). 


3 Cotonou Agreement — Consilium (europa.eu). 
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The research fellows were aged under 40, with a median age just under 35. Though research fellows 
from a variety of disciplines were considered, there was a focus on quantitative skills that would be 
useful in programme and project evaluations. Most applicants were economists, with a strong 
grounding in the application of statistics and econometrics to the social sciences. There was a 
preference for those with or about to complete a PhD. Applicants had to take a test at the application 
stage, focusing on the understanding of, and ability to use, quantitative techniques. The test also 
checked data handling experience, strategic approach and communications. Quantitative analysis, 
while necessary, did not rule out the use of other techniques in deep dives — research fellows were 
encouraged to supplement their work with qualitative and institutional analysis. 


Although female applicants were in a minority, the deep dive programme tried to ensure as much 
diversity as possible in the selection process. Among applicants with similar experience and training, 
women were preferred. Although gender parity was not achieved, the figure of 25% women among 
the research fellows is not insignificant, given the rate of female representation in the economics 
profession generally* and the fact that recruitment was targeted at those from low- and middle- 
income countries. Both factors limited the female applicant pool. 


Table 1: Characteristics of research fellows 


Number Number of Number Median % % sub- % 

recruited/number | recruits with | with years of women Saharan living/working 

applied a PhD in foreign age/work African in sub-Saharan 
economics degrees experience nationals | Africa 


(% of total) 


Cycle1 | 10/128 8 (80%) 8 Under 35/ | 30 100 70 
3.5 

Cycle 2 | 10/158 8 (80%) 6 Under 35/ | 20 100 60 
3.5 

Cycle 3 | 10/189 10 (100%) |7 Under 35/ | 30 90 70 
4 


The final recruits were chosen by a committee consisting of expert advisors, the EIB and the GDN. 
The candidates all had strong academic credentials. There appears to be a strong pool of researchers 
in sub-Saharan Africa with an academic background in quantitative microeconomics, who could be 
suitable to carry out deep dive studies. It was sometimes difficult to choose among the finalists. 
However, the researchers were relatively inexperienced in carrying out independent research, few had 
experience working in or with the private sector, or of analysing private-sector investments from an 
academic viewpoint, and few had experience of independently managing research projects. 


Given the relative inexperience of the research fellows, they were allocated into teams to relieve 
the pressure and enable them to provide support to each other, especially because of the intensity 
of engagement and data work. Most of the research fellows formed teams of two (one worked alone) 
and there were 16 deep dive projects. This fitted with the number of projects funded under the Impact 
Financing Envelope that were at a suitable operational stage for analysis. The pairing provided a good 


4 See https://blocnotesdeleco.banque-france.fr/en/blog-entry/economics-where-are-women. 
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geographical match, as pairs of research fellows were more likely to include at least one researcher 
originating in countries where the projects analysed were located. 


1.2. Research support 


Through EIB funding, the researchers were given a one-year fellowship that included a two-week 
induction course, financial resources to conduct the analysis and access to research support from 
the expert advisors. They received a stipend of €15 000 and a research allowance (for surveys and 
other research-related expenses) of up to €10 000, amounting to €20 000 per project where the 
researchers worked in pairs. 


The principal research capacity support given to the researchers was through five expert advisors, 
who were chosen because of their background in development research, impact assessment and 
evaluation’. Their principal role was to mentor the research fellows during the design, conduct and 
write-up of the deep dives. Each expert advisor was given a notional research fellow allocation, 
although some assignments were shifted after project selection if a particular expert advisor had a 
comparative advantage in the project to be implemented. 


The support from the expert advisors was supplemented by the logistical services provided by the 
GDN and a dedicated team from the EIB Economics Department, with input from the Evaluation 
Department and other EIB services. The EIB team managed the partnership for the Bank, especially in 
making links to EIB clients and acting as facilitators between the researchers and the private-sector 
counterparts. They reviewed documents from a technical perspective and to ensure that EIB 
operational practices and the business models of the EIB clients were appropriately reflected, drawing 
on their professional expertise and roles in the Economics Department and Independent Evaluation 
Division. They provided significant support in editing the studies to a standard acceptable for 
publication as an EIB report, and worked with the researchers to provide readable summaries that 
were circulated as blogs. 


2. How were the deep dives implemented? 


The steps in conducting the deep dives were as follows: project selection; identification of the 
evaluation questions by all stakeholders, taking into account the context of the projects, and 
development of an analytical framework to analyse these questions; write-up and review of the 
research plan including a literature review; data collection and analysis; write-up and review of the 
finished draft deep dive. This section describes each and summarises some of the issues confronted. 


To facilitate the integration of stakeholder experiences and opinions concerning the various aspects 
of the interventions, a short survey was given at the time of writing this book to all the stakeholders 
of the programme, namely the EIB staff involved, expert advisors, research fellows, managers of the 
investment companies who acted as financial intermediaries, and managers of the firms who were the 


5 The five advisors, Francois Bourguignon of the Paris School of Economics (PSE), Shahrokh Fardoust of the College of 
William and Mary, Emmanuel Jimenez of the International Initiative for Impact Evaluation (3ie), Arianna Legovini of the 
World Bank, and Alexandros Sarris of the University of Athens, saw the project as an opportunity to build evaluative 
research capacity and to promote richer interaction between academia and the operations of a development finance 
institution such as the EIB. 
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ultimate recipient of the investment funds. These short surveys aimed at eliciting the opinions of all 
stakeholders involved with respect to the key aspects of the project. These included questions on the 
following: how the ultimate beneficiary firms were selected; who among the beneficiaries experienced 
change as a consequence of the financing and how; types of economic and social value created; the 
thoroughness of the theory of change produced; impressions about whether the various changes 
would have happened in the absence of the intervention; the extent of useful knowledge learnt by the 
firm managers; the research design of the deep dive; the implementation of the research including 
survey design and conduct; the clarity and robustness of the research results obtained; and the type 
of research capacity built. The answers to these short surveys were incorporated into this and other 
chapters of this book. 


2.1. Project selection 


The EIB proposed a preliminary list of projects before recruiting the fellows so that some 
geographical and capacity matching could occur. However, the list was intended to be amended and 
modified depending on the implementation stage of a project, the appetite of funds and companies 
receiving EIB support and implementing the projects to participate in the deep dive exercise, and the 
research potential of each project — that is, the possibility to address relevant and useful research 
questions. The research fellows reported that firms were willing to participate because it would help 
with future expansion and planning (Figure 2). This is roughly aligned with how firms viewed the deep 
dives, as discussed in the next chapter. 


Figure 2: Number of deep dives on the elements that convinced firms to participate, as reported by the 
research fellows 


12 


10 


Research is useful to Research is required by Research will help the Do not know 
firm EIB or investment fund firm with future 
expansion/planning 


Source: Survey of research fellows 
Initial projects were assigned by the GDN to the research fellows before the two-week introductory 
induction. They were required to study them on the basis of available information and to submit a 


provisional research work plan that would be discussed during the induction. These work plans were 
subjected to comments (by the expert advisors, other researchers, the EIB and the GDN). The 
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information provided by that discussion enabled the list of projects to be fine-tuned and allocated to 
research fellows. In the first cycle, the matching problem (based on capacity and geography) was dealt 
with by grouping research fellows in pairs. Given the range of projects under consideration, mentoring 
facilities were enhanced by grouping the expert advisors in pairs for each project, on the basis of their 
own interests and expertise. 


During the first cycle, it took up to three months (Table 3) to select projects and finalise the 
allocations (2.5 months on average). Thus, most research fellows had less than ten months to identify 
the research questions and complete the analysis. 


Most EIB investments analysed under the programme were undertaken via financial intermediaries 
(microfinance institutions, microfinance funds, and private equity and venture capital funds), which 
in turn funded firms (typically small and medium enterprises, or SMEs) in various sectors. The deep 
dive process initially chose the financial intermediaries that were willing to participate. In most cases, 
these financial intermediaries were not directly the subject of the deep dives. Instead, those 
intermediaries, together with the EIB and the researchers, selected firms that they had financed that 
could be interesting targets of the studies. These firms were then asked if they would be willing to 
participate in the deep dive research. They operated in a wide range of sectors, including microfinance, 
information and communications technology (ICT), education and health (Figure 3). 


Figure 3: Sectoral breakdown of deep dives (number of deep dives) 


Pà 


m Communications and IT m Energy m Health 


m Microfinance = Other finance and services = Trade 


The programme aimed to examine a variety of investments rather than providing a representative 
picture of the portfolio under the Impact Financing Envelope®. In most cases, the focus on the 
ultimate recipient of the funds also meant that the operation of the financial intermediaries was not a 
focus of assessment. This was a deliberate choice since various stakeholders, including the EIB, wanted 
to know the effect on environmental and social outcomes of actual investments. 


6 The EIB has an independent evaluation group that examines individual projects, sector-wide policies, programmes, 
partnerships and financing instruments. External stakeholders, such as the European Commission, may also undertake 
evaluations of the mandates or investments that they support. 
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The choice of sectors for the deep dives (Table 2) was guided by practical criteria (willingness to 
participate) and feasibility concerns (for studies that could be completed under the resources 
allocated to the programme). This differed from a sector-based strategy, which could include a series 
of studies in a given sector and deepen learning about what works in different contexts. 


The opportunistic approach also partly explains the wide geographic distribution of the deep dives. 
Twelve of the deep dives were in eight African countries: Benin (one), Cameroon (one), Côte d'Ivoire 
(one), Ethiopia (two), Ghana (one), Kenya (three), Nigeria (two) and Senegal (one). Two were in 
multiple African countries. Two were in the Caribbean (Haiti and Jamaica). None were in the Pacific. 
This precluded any strategic focus on certain sub-regions, but had the advantage of covering a diverse 
group of ACP countries. 
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Country Financial Company to Sector of Key research questions Principal method of assessment Mixed methods | Original data 
intermediary be evaluated economic used? generated? 
activity 

Kenya TLCom Ajua CRM ICT - What are the determinants of CRM software - Theory of change (TOC) No (except the Yes (surveys) 

adoption? - Econometrics (with the use of two | use of theory of 
- What is the impact of CRM software on instrumental variables) change) 
customer volume and sales performance? 

Multiple Direct Currency Finance - In a given African country, how has - Qualitative and quantitative Yes Yes (qualitative 

countries Exchange and other infrastructure been financed (local or hard analysis interviews) 

Fund (TCX) related currency) and why? - Construction of a financial model 
services - How has foreign exchange volatility been? Is it 
costly? 
- What is the willingness to pay for hedging? 

Ethiopia Direct M-BIRR Finance - What is the average stated willingness to pay - Estimation of the willingness to Yes Yes (surveys and 
and other for mobile money use in utility bill payments? pay (using stated preferences on semi-structured 
related - How sensitive to price is the demand for mobile | random given attributes, the interviews) 
services money use in utility bill payments? contingent valuation methodology 

- How effective are non-price factors in affecting | anda probit model estimation) 
the demand for mobile money use in utility bill - Qualitative research (interviews) 
payments? 
Kenya Novastar East | SOKO Project Trade - For artisans, what are the main impacts of - Theory of change Yes Yes (surveys, 
Africa company affiliation on income, skills, access to - Econometrics (t-test, regressions, interviews) 
Ventures healthcare and education, and poverty status? propensity score matching 
- Is there a differentiated impact of SOKO by - Qualitative research 
gender? 
Cameroon Nachtigal Energy - The potential impacts of Nachtigal Hydro Power | - Theory of change Yes Yes (surveys, 


Company investments on vocational education 
and training. 

- Does the renovation or building of school 
infrastructure and the provision of quality 
training equipment lead to increased student 
enrolment and attendance in vocational schools? 


- Econometric estimation 


interviews) 
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Country Financial Company to Sector of Key research questions Principal method of assessment Mixed methods | Original data 
intermediary be evaluated economic used? generated? 
activity 
Nigeria Energy Access | RenSource Energy - What is the profile of Solar_Co’s customer - Theory of change Yes Yes (surveys, 
Ventures base? - Descriptive statistics interviews) 
- What drives some merchants to subscribe to - Econometrics (including using 
Solar_Co solar energy? propensity score matching and 
- How do Solar_Co customers benefit from the endogenous switching regression 
service? (ESR) approaches) 
Jamaica Development | Finance - What is the impact of DBJ loans on microfinance | -Econometrics (instrumental Yes Yes (surveys, 
Bank of and other institutions? variable and two-stage least interviews) 
Jamaica (DBJ) | related - What is the impact of DBJ loans on the squares (2SLS)) 
services performance of SMEs? - Qualitative analysis and review of 
public quantitative documentation 
Ghana Baobab Finance - Has BMC been reaching the poorest women in - Theory of change Yes Yes (surveys, 
Microfinance and other its areas of operation? - Econometrics (average treatment interviews) 
Company related - Has microcredit impacted women’s economic effect, including using propensity 
(BMC) services empowerment in the five districts of BMC score matching and inverse 
operations? probability weighting (IPW)) 
- Has there been a reduction in the poverty level - Qualitative research 
of BMC’s female clients? 
Haiti Microfinance Finance - Did the receipt of an ACME loan stimulate the - Econometrics (using matching) Yes Yes (surveys, 
(ACME) and other expansion of Haitian microenterprises? interviews) 
related - Did it help Haitian microenterprises acquire 
services more fixed business assets? 
- Did it have an impact on the profits of 
microenterprises? 
Côte LMDF Première Finance - Understand the profile of beneficiaries and - Econometrics No Yes (surveys) 
d'lvoire Agence de and other former beneficiaries. 
Microfinance - | related - Assess whether they have benefited, and if so 
PAMF-CI services how, from borrowing (in terms of business 
success, incomes, wealth and subjective well- 
being). 
- Measure customer satisfaction and draw 
strategic and operational lessons to inform the 
design and implementation of operations. 
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Country Financial Company to Sector of Key research questions Principal method of assessment Mixed methods | Original data 
intermediary be evaluated economic used? generated? 
activity 
Nigeria Partech TradeDepot ICT - To what extent has the company’s digital retail - Theory of change No Yes (surveys) 
distribution platform lowered the supply price of | - Literature review 
a select basket of goods for small retailers? - Quantitative methods 
- To what extent do retailers who use the 
platform pass on the lower supply price of goods 
to consumers? 
- Which groups of small retailers benefit from the 
company’s digital retail distribution platform 
(specifically women and young individuals)? 
Benin PAMIGA RENACA/ Finance - Which financial and output market and non- - Theory of change Yes Yes (surveys, 
PAMIGA and other market mechanisms are associated with - Descriptive statistics focus groups) 
related smallholder farmers’ decisions to invest in - Econometrics 
services agriculture in general, and in irrigation - Qualitative analysis 
technology in particular? 
- How do these mechanisms explain differences 
in irrigation technology adoption among 
farmers? 
Multiple Africa Energy Finance - Review the case studies of the use of risk - Theory of change Yes Yes (interviews) 
Guarantee and other mitigation instruments for energy projects in - Literature review 
Facility (AEGF) | related Africa. - Reviews of former case studies 
services - Demonstrate how such instruments improve (impact, costs and so on) 
the viability of energy projects by reducing the - Realisation of a case study: cost- 
financing costs. benefit analysis 
Senegal I&P NEST Health, - Who are the NEST patients? - Theory of change Yes Yes (surveys, 
education - How willing are patients to pay for and to use - Descriptive statistics focus groups 
and midwife-led monitoring and delivery care - Auction methods and field 
related services? Why? - Field experiment experiments) 


- Does the provision of information via short 
stories (that is, visualisation) influence patient 
demand for midwife-led care services better than 
basic information? 


- Qualitative analysis 
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Country Financial Company to Sector of Key research questions Principal method of assessment Mixed methods | Original data 
intermediary be evaluated economic used? generated? 
activity 
Ethiopia Cepheus Finance - What are the main barriers to financial access - Descriptive statistics No Yes (online and 
Growth and other for firms in Ethiopia? How do they influence the - Econometrics (including using field surveys) 
Capital/ related different types of growth-oriented investments? instrumental variable and 
Novastar services - Does a lack of access to finance constrain firm propensity score matching 
growth and investment decisions? 
Kenya Novastar East | poa! internet ICT - How did providing free internet access to - Non-experimental econometrics Yes Yes (surveys and 
Africa schools in Kenya affect student outcomes (including using inverse probability interviews) 
Ventures (including internet skills, ICT skills, school weighting and propensity score 
attendance and access to e-education)? matching 
- Determination of customer satisfaction and - Qualitative research 
related challenges. 


Note: ICT refers to Information and Communications Technology 
Source: EIB-GDN (2019, 2020, 2021) 
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2.2. Defining the research/evaluation questions 


The programme demonstrated that deep dive studies could be valuable to companies and the EIB 
while appealing to the academic interests of researchers and expert advisors with technical rigour. 
This meant that the initial identification of a question by the research fellow was generally followed by 
iterative discussions with the EIB team, the project promoters and the expert advisors. 


Based on these discussions (and in line with the objectives of the deep dive programme presented 
in Box 5 in Chapter 1), the research fellows adjusted the research focus to meet the expectations of 
the EIB and its clients and to fit with the particularities of the business models. In many cases these 
were not initially fully understood by the researchers, who were not experienced with private-sector 
operations. The iterations between the companies and the research fellows helped to fit the research 
ambitions with what would be practically feasible in terms of timeline and data availability. Several 
iterations were often needed, with many of the discussions facilitated by the EIB team. Few of the 
research fellows had experience communicating with private-sector companies, and often presented 
their ideas in a highly technical and academic manner. It was sometimes hard for the private-sector 
companies to see how the research could be of interest to them. The EIB team and, in some cases the 
expert advisors, acted in many cases as “translators” between the young researchers and the private- 
sector counterparts, particularly at the initial stage. 


Table 2 shows the research questions that were the focus of the assessments. Many of the questions 
were of clear interest to the company as well as to the EIB, such as more information about the 
customer base. This spanned a wide range of services, including take-up of microfinance loans, 
midwifery services or use of telecommunications for mobile payments. Collecting this detailed profile 
information, which went beyond the information the project promoters could reasonably collect ona 
regular basis, gave the EIB deeper insights into its social impact, particularly whether the customer 
base served relatively poor and disadvantaged groups. The profile information was also often useful 
for commercial purposes, and in some cases more detailed commercially relevant information was 
collected, including the willingness to pay for services that would affect the companies’ bottom line 
materially, or the degree of satisfaction with the service the company has provided (see Chapter 3 for 
further information about the use of deep dives). 


In many cases it was relatively easy to identify overlapping interests. In most cases, however, the 
research ambitions of the research fellows had to be narrowed to fit with the availability of data, time 
and financial resources. As noted in the earlier chapters, the other stakeholders sometimes had 
objectives that differed markedly from research fellows’ original proposals’. While some shared the 
research priorities initially identified by the research fellows, most had more immediate operational 
questions in mind, which required different research methods. For example, in one of the studies, the 
research fellows initially proposed using satellite data to study the effect on economic activity in the 
area over time. Since this aspect was already being assessed by another investor, the company asked 
the researchers to focus ona relatively small sub-component — the upgrade of training programmes in 
the affected communities — to see if their social investments were helping to improve the lives of 
people in the affected communities. This required a reassessment of the research proposal. 


7 For more than ten deep dives, firms reported that they had accepted to take part in the deep dive exercise because they 
were convinced that it would help later expansion. In four of the deep dives, research fellows had to use the argument that 
the EIB required an evaluation. 
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This process of interaction to identify suitable research questions proved essential to ensure both 
operational and academic relevance for the studies. The average time from the choice of a project to 
the start of relevant research activities was two to four weeks. 


2.3. Development and review of the research plan 


Once the research questions were defined, the research fellows wrote up their approach, defining 
what data and methods were going to be used to address them. This document was then reviewed 
by the expert advisors and the EIB, who typically raised questions about whether the methodology 
selected would enable the research questions to be answered, the likely availability of the data to 
implement the methodology, and the realism of the implementation plan. The research plans were 
typically revised at least once after the initial round of comments. The research fellows were 
responsible for keeping the private-sector companies updated on the research plans. 


A key part of the review centred on recreating the theory of change of the intervention, which was 
key to understanding the pathways through which the companies’ inputs would affect the 
outcomes. A theory of change presents which assumptions are key to ensuring that project inputs will 
lead to stated outcomes. It discusses how inputs, such as materials or staff bought with loan proceeds, 
are combined through activities (production processes) that lead to outputs such as products. 
Assumptions include the availability of factors of production and efficiency of implementation. A 
theory of change outlines how such products are marketed and sold, thus generating outcomes such 
as increased revenues, and impacts on the incomes and well-being of those who benefited from the 
loans. Key assumptions are the demand for the products, the marketing skills of the sellers and how 
they manage the revenues generated. Such a theory of change can be used to identify the key 
assumptions that connect inputs to outcomes and can thus be the subject of the most relevant 
research questions. A theory of change is also important for understanding the reasons behind 
quantitative research findings regarding impact. 


The managers of the companies that benefited from the loans envisaged a logic chain when 
interviewed, and the research fellows reconstructed the theory of change in conjunction with EIB 
staff and the expert advisors. About one-third of the survey responses of the expert advisors 
expressed the view that the project did not have a fully articulated theory of change. A similar 
percentage felt that economic theory could have been used more by research fellows in crafting the 
theory of change. This was an area where the expert advisors showed their value added to the process 
and made suggestions to clarify key assumptions critical to determining outcomes. 


Box 4: How do interactions between research fellows and expert advisors affect the preparation of research 
work? Assessing the impact of SOKO 


The EIB supports investment companies and funds in African, Caribbean and Pacific countries, 
which in turn have invested in local small and medium enterprises. SOKO is a Nairobi-registered 
firm that has been financed by Novastar, an investment fund company based in Kenya and Nigeria. 
It designs ethically sourced fast-fashion jewellery and accessories, which are produced by a growing 
network of low-income artisans in Kenya, and sells the products online to consumers in the West 
through hundreds of fashion boutiques, large established retail chains and direct online channels. 


SOKO's central innovation is its variable cost “virtual factory” of artisans managed by a proprietary 
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mobile technology platform. It uses smartphones to connect independent artisans and global 
customers directly. 


The project undertaken by two research fellows of the GDN and supervised by two expert advisors 
investigated the social impact of SOKO membership in Kibera, a suburb of Nairobi. The initial 
research questions formulated by the research fellows asked whether SOKO membership enhanced 
income and sustainable skills, as well as investments, children’s education, family health and labour 
market participation. The methodology proposed was to compare SOKO members (lead artisans 
who are workshop owners and workers) with a counterfactual group of potential SOKO members 
through a propensity score matching (PSM) technique. 


After a first round of review and suggestions by EIB staff and the expert advisors, the proposal 
was amended. First, with the help of expert advisors, a sound theory of change between the SOKO 
inputs and the impacts was formulated based on human capital theory. In SOKO’s case, the human 
capital investment consisted of on-the-job training for lead artisans and workers. Second, on the 
suggestion of expert advisors, the research questions were modified to be more specific and to 
include the following: the impact of SOKO affiliation on lead artisan and worker income, access to 
health and education, skills, poverty status, issues of income diversification and whether women 
were motivated to participate. 


A survey was designed with the support of the expert advisors in framing the questions and 
administered to the treatment and control groups. Problems arose in achieving the required 
number of interviews for the treatment and control artisans, because the sample frame was 
incomplete. Various techniques were employed to amend this, particularly “snowballing,” whereby 
an artisan was asked for contacts with other artisans. 


The initial method employed for the analysis was comparison of means for diverse variables 
among SOKO- and non-SOKO-affiliated artisans. This method cannot provide attribution for the 
differences, so the expert advisors recommended using econometric techniques. This was done, and 
a propensity score matching methodology was used to select the most similar non-SOKO artisans to 
those who were SOKO members. The result was a sounder analysis of impact on economic and non- 
economic aspects of artisans’ well-being, and a more detailed analysis of the social impact of SOKO. 


As a measure of welfare and living conditions, a Poverty Probability Index was used, despite 
recognised weaknesses. Several other variables thought to be correlated with welfare were tested. 
The end result was that workers with SOKO affiliation seem to enjoy higher welfare levels, and SOKO 
seems to have had a positive social impact on lead artisans and workers. With some caveats, these 
conclusions appear to be valid, although more robustness checks are needed. 


There were other areas where expert advisors contributed, such as the feasibility of methods, the 
adequacy of data and the write-up of the results (Boxes 1 and 2). Some initial drafts overreached in 
writing up conclusions when the methods to establish causality were imperfect. In several projects, 
expert advisors provided guidance on the proper wording of what constituted programme “impacts.” 


An important issue was what method to use. The programme aimed to employ rigorous methods 
using counterfactual analysis. In most cases, however, the so-called “gold standard” of randomised 
controlled trials could not be employed. Only one deep dive implemented a field experiment (Table 3). 
The short time frame, and the fact that many of the supported businesses had been operating for 
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several years and some time had passed since the receipt of support from the EIB or the funds, made 
it difficult to implement a randomised controlled trial. Most deep dives used quasi-experimental causal 
analysis and four relied on correlations to make their conclusions. A key lesson is that deep dives need 
to identify a protocol for what would constitute a “satisfactory” causal analysis under practical 
constraints. 


Table 3: Distribution of methods and cost of research across cycles 


Research | Average Average Number using Number using | Number Average cost 

cycle duration of | duration of randomised other quasi- unable to of research 
project defining controlled experimental do causal including 
selection research trials techniques analysis researcher’s 
(months) questions stipends (in 

(months) thousands of 
euro) 

Cycle 1 3 3 0 2 3 25.0 

Cycle 2 3 2 1 4 (0) 26.5 

Cycle 3 2 1 (0) 4 1 28.5 

Total 2.5 2 1 11 4 27.0 


Another issue was the availability of data to conduct the analysis. Expert advisors often warned about 
the difficulty that researchers would have, and they could not assume that data would be readily 
available. Primary data collection had to be worked into the budget and the available timeline. 
Researchers had to explore all possibilities to exploit secondary data. This is detailed in the next 
section. Box 2 provides examples of how criteria were used to adjust research designs. 


Box 2: Alternatives to complete counterfactual analysis when experimental or quasi-experimental methods 
are unfeasible 


In the deep dive of the TCX investment, the original research design, besides providing 
information about major energy projects involving foreign providers, was an elaborate economic 
modelling of the social willingness to pay for hedging against exchange rate volatility in a power 
plant operated by a foreign provider in Tanzania. Application to real data was disappointing, 
because Tanzania's exchange rate had been relatively stationary, with little short-run volatility over 
the recent past, whereas domestic prices showed limited fluctuations around a stable inflation 
trend. Hedging proved of little interest based on past exchange rate behaviour. The expert advisors 
guided the focus away from present conditions so that the deep dive evolved towards a full 
simulation analysis of the social willingness to pay for hedging against varying levels of exchange 
rate volatility. This analysis provided the hedging company with an estimation of the full demand 


function for its services and the public energy supplier with an instrument that would help evaluate 
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the contracts offered by the hedging company, depending on key policy parameters like the extent 


of the pass-through of exchange risk to end users, risk aversion or time discount rate. 


Source: EIB-GDN (2021) 


Researchers also had to be aware of the limited budget, which allowed for a maximum of two field 
trips. Research fellows were invariably overoptimistic in their plans and had to adjust them to fit reality. 


2.4. Data collection and analysis 


All the deep dives had to supplement the data provided by the companies with primary data 
gathered from surveys and interviews. Survey design, data collection and data entry collectively took 
over 14 weeks on average (Figure 4). The survey questionnaires took an average of 25-35 minutes to 
conduct. In some cases, administrative data were helpful to identify basic client characteristics (such 
as data on gender from the client bases of telecoms providers), but did not have the requisite 
information on outcomes needed to address the research question. Deep dives invariably needed to 
build in a significant time period for surveys, interviews and focus groups to amass a sufficient database 
for analysis. The research fellows were generally able to achieve this within their time and resource 
constraints, despite the further restrictions imposed by the COVID-19 pandemic during 2020-2021. 


Figure 4: Average number of weeks by phase 


Survey design 


Survey conduct 


Data analysis 


Presenting and explaining the results to firm 
stakeholders 


Entering and cleaning data eel 


Source: Survey of research fellows 


Expert advisors and the EIB were able to comment on most of the questionnaires and survey guides. 
Some research fellows conducted interviews, but most employed local survey companies. Some 
conducted in-person interviews while others carried out interviews by phone or online, particularly as 
a result of the COVID-19 crisis. Research fellows reported generally being satisfied with the data 
collection. Pilot surveys were done in some cases. Expert advisors were not usually able to check on 
the quality of the data, except when the research fellows flagged anomalies. Expert advisors assumed 
that the research fellows had done the requisite due diligence to assure reliability. 
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Data analysis, which lasted just over ten weeks on average, was undertaken by research fellows. 
They often sent work in progress to be reviewed by expert advisors. An initial report describing the 
data was usually circulated prior to a draft final report that tested the research hypotheses. The 
research fellows were generally responsive to addressing the issues raised during the reviews, 
according to the expert advisors. The research fellows reported that they received useful feedback 
from the expert advisors. 


The EIB and GDN recognised the importance of active engagement with the companies that were 
being assessed. At the recommendation of the EIB, a “data note” was produced after the data 
collection phase and before the final reports. This kept momentum, ensured companies were properly 
informed about the status of the research, and was an opportunity to share preliminary insights of the 
research. 


2.5. Review 


The draft reports were reviewed by the expert advisors and the EIB. This progressed faster when the 
initial data descriptions had been shared earlier. The review process generally took a relatively short 
time, usually four to six weeks, thanks to the responsiveness of the research fellows. 


Most of the issues raised by the expert advisors and the EIB during review were technical, many 
dealing with the robustness of the causal claims in the report. Data limitations (such as the lack of an 
adequate baseline or control) meant some final reports had to include substantial caveats. Many 
comments concerned the logical structure and readability of the report. The reviews required an 
average of two to three iterations to produce a draft final report. 


After the technical review, companies were asked to comment on the final report: many provided 
suggestions or factual corrections. The research fellows reported these comments to focus on 
clarifying and simplifying the write-up to make it understandable to a general audience. The firms’ 
reactions are detailed in Chapter 3. 


3. What was produced by the deep dives? 


Table 4 lists the final reports that have been shared and discussed with the companies and other 
stakeholders. These are the outputs of the deep dives. Chapter 3 reports on the results of that sharing. 
Here we consider the products for each research cycle in terms of reports and databases and discuss 
whether and how they met initial expectations. 


16 research studies were completed: ten of these reports are publicly available and six are expected 
to be shared publicly by April 2021. Most of the reports reached a conclusion of potential commercial 
value to the relevant businesses. These included information on the nature of their client base and the 
demand for their services. 


Many of the reports addressed the social returns. For example, there was much interest in findings 
related to the client base, particularly the income groups that the company reached. This sometimes 
led to considerations of how to reach less wealthy market segments (see Chapter 3 on the uses of the 
deep dives). 
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Internet in Low-income Areas of 
Kenya 
Charles Yaw Okyere 


Report title and author names Research | Databases generated | Journal article | Main conclusion 
cycle (availability and prepared 
storage of data sets) 
Potential Impacts of the Africa 1 No No - Risk mitigation instruments have been used to mitigate energy investment risks in Africa 
Energy Guarantee Facility with success. 
(AEGF) - These instruments need to be provided alongside other forms of support such as 
Oluwasola Omoju technical assistance. 
- Major demand-side factors limit the adoption of these instruments. 
- Evidence from similar instruments for small and medium enterprises points to 
significant positive effects. 
- The AEGF has good potential to reduce financing costs, which would have a major 
impact on development indicators. 
M-BIRR: Mobile Banking in 1 Yes Submitted - Determination of the social value of providing utility bill payments by mobile money. 
Ethiopia - Proof that the willingness to pay is higher than the marginal cost and that there is 
Yesuf Awel & Eleni Yitbarek potential business demand. 
- Determination of technical, legal and regulatory needs. 
Premiére Agence de 1 Yes Submitted - PAMF is actively contributing to financial inclusion in some of the poorest areas of Côte 
Microfinance (PAMF): d’ Ivoire. 
Promoting Financial Inclusion in - Clients are highly satisfied with the products and services. 
Côte d'Ivoire - Most loans invest in business activities. 
Uwingabiye Gloria & Guylaine - Successive borrowing has a positive impact on the economic and financial performance 
Nouwoue of microenterprises. 
- No evidence could be found of an impact on longer-term fixed asset accumulation. 
- Women make up over half of the client sample, although the impact on business 
outcomes is less pronounced for female entrepreneurs. 
poa! internet: Community 1 Yes Submitted - Improved access to unlimited data has changed the way students and teachers use the 


internet. 

- Students in poa! internet’s partner schools are more likely to benefit from ICT training 
during school hours than their peers in comparable schools and make more use of the 
internet for private purposes, including accessing educational content. 

- There was no evidence that the internet was being used to enhance learning outside of 
ICT training. 

- Improvements in school ICT infrastructure may be needed to translate better internet 
access into enhanced student achievement. 
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oriented Investments by 
Manufacturing Enterprises in 
Ethiopia 

Muhammed Abdella Usman & 
Seid Hassen Mohamed 


Report title and author names Research | Databases generated | Journal article | Main conclusion 
cycle (availability and prepared 
storage of data sets) 
Irrigation Technology Adoption 1 Yes Submitted - Farmers in rural Benin find it difficult to access finance to improve agricultural 
and Microfinance in Rural productivity (although most respondents are financially included, only a minority have 
Benin: RENACA’s Agricultural accessed a formal loan). 
Loan Programme - RENACA is contributing to financial inclusion, in particular by providing investment 
Mariam Sangaré & Bassirou Sarr assets and working capital (fertilisers, seeds and so on), which appears equally important 
to farmers. 
- Lack of collateral reduces the ability of farmers to access finance and to invest. 
- Farmers who rent or borrow land indicated a lower willingness to invest in their 
activities and adopt advanced technologies, including irrigation equipment (they invested 
in transport, which allowed them to get perishable products to markets in time but did 
not improve productivity in the long run). 
Impact of Customer 2 Yes Submitted - Information technology infrastructure, firm size and informal competition are positively 
Relationship Management associated with the adoption of CRM software. 
Software on Firm Performance - CRM software has a positive and significant impact on two measures of firm 
in Kenya performance (customer volume and sales). 
Abdi Yuya Ahmad & Laura - Government investment in ICT infrastructure could play a critical role in fostering firm 
Nelima Lalampaa performance and overall economic growth in a sound investment climate. 
Provision of Quality and 2 Yes Submitted - Clients are middle-class women (although few are among the extremely poor), including 
Affordable Maternal Health those without health insurance coverage and those working in the informal sector. 
Care in Developing Countries: A - Providing information via visualisation raises both willingness to pay and use of the 
Case Study of NEST in Senegal service more effectively than providing basic information. 
Hamidou Jawara & Gilles - Overall, participants are satisfied with the services offered but highlight some possible 
Quentin Kane improvements. 
Finance Access and Growth- 2 Yes No - About 27% of the surveyed enterprises obtained a bank loan, whereas 10% relied on 


informal sources. 

- Bank collateral requirements are one of the main causes of loan applications being 
rejected. 

- A large share of debt money is directed to investments aimed at improving products 
and industrial processes. 

- Enterprises rarely obtain finance from private equity or venture capital firms 

- Overall, firms that use debt capital show a capital growth rate at least 7.6% higher than 
that of firms with no debt capital. 
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Report title and author names Research | Databases generated | Journal article | Main conclusion 
cycle (availability and prepared 
storage of data sets) 

Microfinance Loans, Women’s 2 Yes Submitted - Data suggest that BMC clients experience lower poverty. 

Economic Empowerment, and - This is consistent with focus group discussions with control respondents and BMC 

Poverty: A Case Study of clients. 

Baobab Microfinance Company - Clients are not always among the extremely poor, and female clients are more 

Franklin Amuakwa-Mensah & economically empowered. 

Edgar Cooke 

Understanding the Sustained 2 Yes Submitted - On average and across different socioeconomic outcome indicators, worker artisans, 

Impact of SOKO on Artisans in including female artisans, affiliated with SOKO do better than their non-SOKO 

Kenya: Empirical Evidence from counterparts. 

Kibera - In SOKO and non-SOKO workshops, female worker artisans are more vulnerable to 

Timothy Kinoti & Soazic Elise poverty than male worker artisans. 

Wang Sonne 

The Impact of Microfinance on 3 Yes No - Receiving subsequent loans from ACME had some positive impact on Haitian 

Business Investment and businesses. 

Outcomes in Haiti - There was a positive impact on business expansion through a different set of outcomes. 

Boaz Anglade - The firms that had received more than three cycles of loans had a higher probability of 
hiring new workers in 2018, a higher probability of opening new branches, and a higher 
likelihood of increasing merchandise stock as a way of expanding their business. 

Development Bank of Jamaica 3 Yes No - The credit amount received had a positive impact on the performance (particularly 

Jacob Novignon profits) of microenterprises and small and medium enterprises. 
- The impact of the loan amount on performance was significant for firms that received 
the loan at least one year before, those that were older (over 40 years) and those owned 
by males. 

Impact of Rensource Solar 3 Yes No - Subscription to Rensource solar energy has a positive impact on monthly earnings, sales 

Energy on Subscribers in volumes and merchant profits. 

Selected Markets in Nigeria 

Esther Leah Achandi & Enoch 

Owusu-Sekyere 

Cost of Funding Infrastructure in 3 Yes No - TANESCO can record positive net benefit from hedging starting from the effective date 


Hard Currency 
Nnamani Alexander Uchenna & 
Kebba Jammeh 


of an off-take agreement and this benefit increases with the lifespan of the power 
purchasing contract. 
- The results reveal TANESCO’s low willingness to pay for currency hedging. 
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Report title and author names Research | Databases generated | Journal article | Main conclusion 

cycle (availability and prepared 

storage of data sets) 

The Impact of Digital Consumer 3 Yes No - ShopTopUp users internalise lower supply prices to reduce consumer prices for selected 
Goods Distribution on the Small products. 
Retail Sector in Nigeria: The - The usage of ShopTopUDp is associated with a reduction in both the gross profit margin 
Case of TradeDepot and the duration of product stockouts. 
Matthew Townshend & Jaison 
Chireshe 
Deep Dive Report on the 3 Yes No - The results show that the activities of Nachtigal Hydro Power Company were unable to 
Impacts of Nachtigal Hydro improve enrolment but increased school attendance among students. 
Power Company Investments on 
Vocational Education and 
Training in Cameroon 
Nantongo Mary Gorret & 
Mireille Ntsama 


Source: EIB-GDN (2019, 2020, 2021) 
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It is difficult to say whether the conclusions that had commercial and social ramifications can be 
generalised more broadly for development. All of the studies are of interest to those who have a 
specific stake (investors, companies and firms whose operations are being evaluated). Moreover, some 
of the conclusions will be of interest to other investors who may wish to consider the social impact of 
investments in the same sectors. The expert advisors felt that six of the ten studies completed in cycles 
one and two could have results that would be of interest to the wider development community 
because of their generalisable findings or methods. The deep dives were implemented in a varied set 
of contexts and were not case studies purposefully selected to provide generalisable conclusions. 
Applying the conclusions to other contexts would need to be done carefully to ensure that the 
comparisons are appropriate. 


Expert advisors felt that the final outputs were the best that could be achieved under the 
circumstances — the relative youth and inexperience of the research teams, the fact that they were 
“parachuted” into ongoing operations, and the time and budget constraints of the process. The work 
has been thoroughly reviewed. Eight of the papers from the first two cycles have been scrutinised by 
blinded external referees, and a special issue of the Journal of Development Effectiveness will publish 
about half of the deep dives. As of March 2021, most of these papers are at the “revise and resubmit” 
stage. If enough papers are accepted for a special issue of the journal, this is a good indicator that the 
overall quality of scholarship meets academic standards. 


Another principal outcome of interest was the building of research capacity. A survey of the research 
fellows at the conclusion of the project showed that the following were most valued: 


e Methodological skills: many research fellows felt more confident and have been using the 
methodology in their projects or teaching. 

e Soft skills and communication: some research fellows feel more confident in their 
relationships with colleagues and other stakeholders (including private-sector firms and 
companies). 

e Completing a research project from beginning to end: some research fellows learnt a lot 
about the different steps, and now feel more confident and better able to guide students. 


Most of the research fellows are continuing their careers as researchers. 


The EIB, expert advisors and GDN largely agreed with the research fellows’ conclusions. If properly 
supported — in terms of budget and mentoring — relatively young researchers at the beginning of their 
careers can produce reports that are relevant to many impact investing stakeholders and are rigorous 
enough to meet academic standards. Implementing such deep dives is a powerful way to build the 
capacity of talented young researchers. Substantial support was required to make these studies a 
success: a programme of deep dives structured in this way would likely be attractive only to 
implementers with an explicit mandate or objective of capacity building. 


4. Main lessons 


What were the main lessons learnt about how to conduct these types of programme? This section 
draws on the findings of this chapter. 


First, deep dive research must consider, from the outset, the general lack of existing data. The 
research should make the best possible use of data available. Deep dives may be easier if monitoring 
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data collected and stored by project promoters are available (respecting relevant confidentiality 
concerns); if the projects assessed are operationally advanced enough to generate data on outcomes 
and impact; and if the opportunities offered by new data sources and data analysis techniques (for 
example, machine learning) can be exploited. However, in most cases researchers will need to plan 
and budget for primary data collection. 


Second, diverse perspectives and needs must be considered and addressed. Company 
stakeholder/investee engagement is key. The buy-in of the small business, for example, that has 
secured financing is essential to ensure successful cooperation and data sharing. Early and frequent 
interactions between companies and researchers are needed to build trust. 


Researchers, on the other hand, have careers to build and manage and most are aiming for academic 
positions. When studying development impact, they want to mobilise their academic knowledge and 
instruments to investigate economic and social returns to a given investment and to produce academic 
studies recognised by their peers. Researchers and expert advisors tend to focus on this and on building 
local capacity. EIB operational staff want to determine the likelihood that their project will be a success, 
and what the implications might be for similar future investments. Investment funds and firms do not 
have time to spend on studies that do not bring them useful business-related insights. 


An ongoing issue is how to formulate research questions that will be most relevant to the investment 
funds and the companies in which they invest. Defining the right research questions implies a 
productive tension between research objectives driven by academic ambition and feasibility, and 
firms’ operational concerns driven by practical relevance and importance in terms of business and 
financial objectives. As discussed in Chapter 1 (aspirations 1 and 2 in Box 5), deep dives should align 
stakeholders’ interests. That tension is at the core of the connection between research and operations, 
and the deep dive programme has explored the connection. 


Getting stakeholders to allow and actively participate in projects was a priority, especially given the 
innovative nature of this first deep dive exercise, so the choice of projects was relatively 
opportunistic. To broaden their contribution to impact measurement, it will be useful to focus several 
deep dives on the same sector and to choose the sectors strategically. Another limitation of ensuring 
meaningful stakeholder buy-in is that the choice of projects may be liable to self-selection bias 
(volunteering projects that are more likely to be successful for a deep dive). This was not an issue in 
our programme given the objective of demonstrating the potential of a research-based approach to 
assessing impact, but should be considered when scaling up the approach. 


Third, researchers must be flexible about research approach and methodology (aspiration 3 in 
Box 5). Researchers, especially those with less experience in operational matters, can allow the 
research methodology and related empirical techniques to drive the research question, indicating a 
need to develop capacity in defining issues of interest and shaping policy or operationally relevant 
research questions. Acquiring such flexibility was an important aspect of the capacity that research 
fellows built throughout this programme. 


One way to ensure that technique does not overshadow relevance is to develop a clear analytical 
framework to clarify a theory of change that identifies the assumptions used in linking investments 
to outcomes. Such an initial framework was available for the majority of the projects and would have 
helped focus the work in all others. Another approach is to be flexible on technique. It was challenging 
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to choose an appropriate methodology for the evaluation work, knowing there is no “gold standard” 
and that conditions for a methodological approach are not always met. There may be a trade-off 


1” 


between technical “purity” and feasibility when studying an operationally relevant research question 


in a time- and process-bound context. 


Fourth, conclusions must be tailored to the limitations of the analysis. While the research fellows 
could conduct technical analysis of the data, it was more of a challenge to draw firm conclusions that 
are not generic and add value to the global community, as well as being operationally relevant. Some 
conclusions that are useful operationally may not seem notable in terms of changing the broader 
development discourse. Other conclusions may not be deemed sufficiently operational by the 
company clients, financial intermediaries or the EIB. Researchers must be realistic about the 
conclusions they can draw and what is justified by the analysis and data. 


Fifth, reports must be communicated in the correct form for accountability, learning and promotion 
(see the three functions of evaluations in Chapters 1 and 3). The importance of this dimension can be 
underestimated. EIB staff had to invest a significant amount of time to make the studies fit for 
publication and to produce comprehensible summaries that could be shared with stakeholders or as 
blogs. Although they received some communication training, the research fellows struggled to connect 
with non-academic audiences. This area of capacity building needs greater weight in future 
programmes. 


Sixth, building research capacity is a slow process (objective 3 in Box 5). A limited amount of technical 
capacity could be built in the short time period available. The preparatory seminar on methods and 
research design helped but could not prepare research fellows for deep technical embedding. The 
deep dives exposed the research fellows to new areas or engaged them further in familiar areas, 


|” 


exposing them to operational constraints in the “real” world. The research fellows and expert advisors 
noted that capacity had been successfully built in terms of discussing research plans with non- 
researchers (companies) and negotiating appropriate research questions (that met academic 
standards and other stakeholders’ needs and interests). The induction seminar included two days of 
communication training, which exposed research fellows to some of the challenges of negotiating a 


work plan with third parties, and eventually framing and reporting their results. 


Seventh, the costs of the deep dive were neither insignificant nor prohibitive and were lower than 
those of typical impact evaluation work. Beyond the production of deep dives, researchers from low- 
and middle-income countries had the opportunity to apply their technical expertise to real-world 
problems and gain experience in dealing with practitioners and policymakers, especially in the private 
sector. In many cases, the EIB, the firms and the intermediaries judged many of the findings useful. 
Despite the budget and time limitations, this exercise showed that, with proper management, research 
can productively combine operational concerns and technical rigour. 
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Summary and messages 


Deep dive impact studies address questions about the social and environmental outcomes 
generated by an investment, and can contribute to organisational accountability, learning and 
communication objectives. 


Drawing on a framework for evaluation use, this chapter sets out four ways in which the European 
Investment Bank (EIB) and Global Development Network (GDN) programme deep dives were used. 


First, the EIB and its private-sector partners used the studies as an input into future tactical and/or 
strategic planning (instrumental use). Some companies used findings to make or validate key business 
decisions, mainly related to marketing and operations functions. Investors (the EIB and fund managers) 
primarily used deep dives to verify and evidence their impact thesis, and for promotional purposes. 


Second, the deep dives laid the groundwork for building knowledge about the effectiveness of 
private-sector investment in Africa, the Caribbean and the Pacific (conceptual use). While deep dives 
did not have a thematic focus, with over 15 individual studies completed, they can be considered a 
body of knowledge that can be used and further developed over time. 


Third, the deep dives fulfilled the requirement to do an impact study, therefore providing a proof of 
concept that more rigorous data collection and analysis approaches are suitable in the context of 
private-sector-focused impact measurement (symbolic use). 


Fourth, the deep dives provided an opportunity for the EIB and other stakeholders to learn whether 
this was a useful approach and how best to implement it (capacity building use). The deep dives 
generated replicable process knowledge on aspects of impact measurement, particularly how to arrive 
at a plausible but practical approach to the question of causal change. 


This chapter addresses whether, and to what extent, knowledge resulting from the deep dives 
conducted by the EIB-GDN programme was used’. The value added for researchers is covered in 
previous chapters, so the focus here is on use by the companies subject to deep dive research and the 
investors that provided them with capital, including departments within the EIB. 


The chapter has three sections. First, we introduce a conceptual framework to understand possible 
deep dive uses and related user needs, drawing on literature from the field of evaluation. Second, we 
present empirical data showing how deep dive information was used, based on stakeholder interviews. 
Third, we summarise the factors shown to influence deep dive use. 


2 Deep dives comprise the set of studies produced under the EIB-GDN Programme in Applied Development Finance. 
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1. A framework for deep dive use 


The EIB-GDN programme deep dives were impact studies — the initiative was a systematic exercise in 
applying rigorous research techniques to understand the social and economic impact of private-sector 
investments. This section begins with the intended use of deep dives as an important starting point to 
see whether they met user needs. To achieve this, we have developed a framework for deep dive use 
that builds on the extensive literature on evaluation use. 


While deep dives are not evaluations, they fulfil an evaluative function in that they address 
questions of the effectiveness and impact of a given investment. They meet four out of the five 
criteria proposed by O’Flynn and Barnett (2017) for a more evaluative approach to understanding 
social impact as follows: impact (what is the effect on society and the environment?); differential 
impact (who benefits and who does not?); accountability (for impact to different stakeholders); and 
plausible causality (has the investment made a difference, and if so, how?)?. Evaluation literature 
offers the richest set of documents and discussions related to the topic of use. 


Although there is strong recognition “that use is central to the reasons for doing evaluations” 
(McDavid et al., 2013), there is little academic consensus on a formal definition of evaluation use. A 
functional definition emerges from the work of Alkin and King (2017) that “evaluation information 
must have consequences of some sort — hence, use.” They argue that use means examining the “extent 
of actions taken or thoughts crystallized as a result of an evaluation” (ibid.). Three key messages from 
evaluation literature can inform our understanding of deep dive use. 


First, use is in the eye of the beholder. Understanding use means understanding user needs and the 
context in which deep dive information can be deployed. This requires a focus on primary users: “the 
stakeholders most closely associated with the program who are potentially in a position to make 
decisions based upon the information” (Alkin and King, 2017). The concept of utilisation-focused 
evaluation (UFE) developed by Michael Quinn Patton stipulates that evaluations should be planned 
and conducted in ways that enhance the likely utilisation of both the findings and the process itself to 
inform decisions and improve performance (Patton, 2012). Utilisation-focused evaluation identifies 
the intended users and documents their primary intended uses. 


Second, a distinction may be made between two dimensions of use. Consequences may arise from 
the results of the evaluation and the conduct of the evaluation*. Use may therefore be derived from 
“evaluation findings as well as from the process, that is, the actions involved in engaging in an 
evaluation” (Alkin and King, 2016). 


Third, a broad definition of evaluation use should take into account the actions or decisions 
concerning the subject of the study following an evaluation (direct use), and the wider indirect 
effects the evaluation may have on other interventions, strategies or on organisational behaviour. 
This latter category has been labelled “evaluation influence,” with academics having long noted the 
potential for evaluations to have a wider impact beyond the individual programme under review 
(Weiss, 1972). 


3 The other criteria being “aggregation” (O’Flynn and Barnett, 2017). 
4We use the term “use” to also cover aspects of evaluation “influence” resulting from the findings of the deep dive and the 
process of taking part in a study. 
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Figure 1 outlines a conceptual framework to understand deep dive use. This distinguishes between 
different stakeholders in the deep dive value chain (“who”); the reasons for conducting a study 
(“why”); the types of deep dive information that can be useful (“what”); and the ways in which this 
information was used in practice (“how”). The remainder of this section explains the framework with 
reference to the EIB-GDN programme deep dives and relevant concepts in the fields of evaluation and 
impact investing. 


Figure 1: A conceptual framework to understand deep dive use 
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Pillar 1: Who are the intended primary users of the deep dives? 


The EIB funded the deep dive programme to deepen understanding of the impact being generated 
by the Impact Financing Envelope. The EIB and its shareholders are therefore both a commissioner 
and a client (user) of deep dive information. Each study took place on a project under the EIB’s Impact 
Financing Envelope for Africa, the Caribbean and the Pacific, focusing on a particular investment made 
through one of the envelope’s four financing instruments. 


Private-sector companies (investees in the terminology presented in Chapter 1) were the subject of 
the deep dive research and expected to benefit from participation. For companies supported via 
financial institutions (such as banks offering microfinance) or through investment funds (such as 
venture capital or private equity funds), these intermediaries and fund managers were also potential 
users. Collectively, we term these “private-sector partners.” 


The two user groups — the EIB and private-sector partners — are far from homogenous. Within the 
EIB, different directorates and departments were involved in the deep dives. The private-sector 
partners varied in terms of enterprise size and stage of business cycle. More granular “use pathways,” 
reflecting these different institutional priorities and perspectives, are explored in later sections. 


Pillar 2: Why carry out a deep dive study? 


The EIB, like other international financial institutions, monitors the extent to which its work delivers 
development impact. As explained in Chapter 1, the deep dives complement the impact measurement 
framework of the EIB to assess whether a given investment — or a part thereof — generated a specific 
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impact as intended, using rigorous research methods. The value of deep dives resides in being able to 
provide high-quality evidence on “what is working and what is not” to the EIB. The findings of deep 
dive studies are aimed at helping the Bank to enhance its contribution to sustainable and inclusive 
development, as well as to enhance its capacity for impact studies. 


From a private-sector perspective, the range of motivations and intentions for engaging in deep 
dives can be captured by proxy, using results from an annual survey run by the Global Impact 
Investing Network. The “reasons for measuring and managing impact” given by respondents to an 
annual investor survey include to better understand impact; to proactively report to key stakeholders; 
to manage or improve impact; to create business value; to communicate impact for marketing and 
fundraising; and because of requirements to measure and report (Global Impact Investing Network, 
GIIN, 2020a). 


For both user groups, a “useful distinction can be made between evaluation done for learning and that 
done for accountability” (Loud and Mayne, 2014; see also Chapter 1, Section 4.1). 


An accountability objective for the deep dive focuses on the need to report “fairly and accurately on 
performance results vis a vis mandated roles and/or plans” (OECD, 2002). Another way to put this is 
measuring in order to “prove” impact. According to the Rockefeller Foundation, the aim is to 
understand whether the short-term changes and long-term effects are occurring in ways that were 
anticipated (Rockefeller Philanthropy Advisers, 2020). A further distinction can be made between 
upward accountability — towards funders and investors — and downward accountability towards 
affected populations. 


A learning objective uses the deep dives as a basis to enhance the relevance and quality of future 
actions. Impact investors are trying to prove impact and, much like financial performance, are 
interested in increasing it over time (Rockefeller Philanthropy Advisers, 2020). This objective focuses 
less on the historical results of the project, and more on the conclusions, recommendations and 
lessons that inform forward-looking action. It covers content learning, dealing with the issue or theme 
at hand, and process learning, such as on how to conduct a rigorous study. 


Finally, deep dive studies also contribute to objectives related to communication. Communication 
cuts across both learning and accountability purposes, and often serves a purpose by disseminating 
either specific (that is, on individual studies) or more generalised information (synthesised across 
studies) to internal and external audiences, serving purposes such as institutional promotion to 
funders or public information campaigns. 


Pillar 3: What types of deep dive information can be used? 


Each deep dive generates different types of information, which are used in different ways. Academic 
and business literature make a distinction between data and insights (Brown, 2020). 


Data are facts and statistics that are usually in the form of numbers and text. Based on a typology by 
New Philanthropy Capital, each deep dive can be considered to have covered up to six distinct types 
of data (Noble, 2017): 


e User data — characteristics of the affected populations, such as socio-demographic data (for 
example, age and location). 
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e Engagement data — extent to which people use or interact with products and services, such as 
uptake (for example, number of loans). 

e Feedback data — what people think of the products and services, such as satisfaction data (for 
example, net promoter score). 

e Intermediate outcome data — changes in user knowledge, attitudes or behaviours, such as 
financial literacy (for example, number reporting being able to better manage finances). 

e Long-term outcome data — mental, physical and economic well-being, such as financial well- 
being (for example, change in savings/debt). 

e Attribution data — the difference made by the intervention, such as degree of change in 
financial well-being (for example, extra savings that would not have occurred in the absence 
of the intervention). 


Insights are generated by using the data to accumulate knowledge and drawing conclusions, and 
then using them to influence decisions and drive change. Insights refer to the specific findings and 
recommendations of each deep dive study, as well as the knowledge built by engaging in a deep dive. 
From a user perspective, insights need to be feasible — that is, geared towards probable future 
activities. Insights also need to be contextually appropriate and grounded in the business reality of the 
subject of the deep dive. 


Pillar 4: A typology for understanding deep dive use 


A utilisation-focused approach looks at the real and specific ways in which evaluations have been 
used, rather than their abstract and theoretical uses. Experience from evaluations shows that people 
tend to use information in four broad ways: first, to make or validate decisions; second, to establish or 
alter attitudes; third, to justify the existence of a project; or fourth, to build an individual’s or an 
organisation’s evaluation capacity (Alkin and King, 2016). In evaluation jargon, each deep dive 
corresponds to four different categories of use: instrumental, conceptual, symbolic and capacity 
building (Box 1). Each of these use typologies has potential to form part of the “feedback loop” that 
uses the results and lessons learnt from the study to inform organisational strategy and/or future 
project design. 


Box 1: Four ways in which deep dive information can be used 


Instrumental use involves turning deep dive information into implications for action. This is 
perhaps the clearest and most straightforward category, for example where deep dives are used to 
improve projects. Instrumental use does not necessarily involve taking direct action, rather it means 
a “serious discussion of findings” and a consideration of the recommendations (Cook and Pollard, 
1977). The result of these deliberations can be taking or not taking action with regard to a future 
decision or justifying or reversing a past decision (Alkin and King, 2017). Instrumental use takes place 
when deep dive results are “considered seriously by persons in a position to act on our information 
if they choose to do so” (Hendricks, 1990). There are two types of potential action. 


5 The EIB distinguishes between the question of attribution (the difference that can be attributed to the service or product 
in question) and contribution (the contribution of the EIB finance to achieving that change). 
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First, measurement is one of the four core characteristics of the practice of impact investing (GIIN, 
2020b). Investors have a commitment to report on the social and environmental performance of 
investments, which requires underlying businesses to measure and report on their own social, as 
well as financial, performance. Marketing uses — such as including the findings of deep dives in pitch 
decks or promotional material — is included here, as information is being turned into a concrete “call 
to action” for fundraising. 


Second, knowledge and insights from the deep dives can be used to fuel data-driven decision- 
making (GIIN, 2016). From an EIB perspective, this can be to manage overall impact performance 
and inform future portfolio construction under the Impact Financing Envelope. From a private- 
sector perspective, learning can inform business decisions and investment approaches “in a way 
that can create tangible business benefits, such as better understanding customers, streamlining 
deal sourcing, or improving interactions with important stakeholders in the communities in which 
they invest” (GIIN, 2016). 


The conceptual use of deep dives enhances knowledge about the type of project under study in 
order to influence thinking about issues in a general way on a particular topic (Mayne, 2014). 
Information can be used internally around a thematic focus or externally to help existing and 
aspiring investors who are working on related impact issues, including by adding to a global evidence 
base (Rockefeller Philanthropy Advisers, 2020). 


Symbolic use fulfils a requirement to do evaluation or to show support for an intervention area. 
This is also called “tokenistic use,” but this term can hide the value inherent in conducting an 
evaluation exercise. The act of carrying out impact evaluation can be important in contributing to 
an impact measurement culture and signalling that an organisation is serious about impact. 
Literature shows that positive changes in organisational behaviours and practices can result simply 
from the plan to carry out an evaluation, independent of the findings (Mayne, 2014). 


A capacity building use comes through engaging in a deep dive process and obtaining a better 
understanding of when, how and in what ways more academic research techniques are suitable 
to measure the impact of private-sector investments. This contributes to the overall management 
of the organisation through institutional learning and development (Alkin and King, 2017). 


The conceptual framework rests on a number of assumptions. First, that the deep dive is of 
reasonable quality. The study needs to be technically adequate and credible such that “its findings 
warrant using” (Alkin and King, 2017). If technical aspects of the deep dive have not been conducted 
properly — for example, incorrect statistical analyses — this can be referred to as a “misevaluation.” 
Misevaluation can also take place when the deep dive researchers fail “to understand the [...] context 
properly and therefore misdirects the evaluation” (Alkin, 1990). 


Second, “use” is framed only in positive scenarios. There are potential uses of evaluations that seek 
to deliberately misrepresent or unethically exploit impact data. For example, companies might break 
data protection and confidentiality promises to contact customers who have given them negative 
feedback, or to cherry-pick the best findings to promote on their website. The field of evaluation calls 
this “misuse;” however, it is not considered further here, mainly due to difficulty in uncovering such 
practices (Mayne, 2014). The “use continuum” extends from use to underutilisation to non-use. 
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Third, instrumental use assumes that the deep dive was one of many sources of information 
considered when making a decision. It does not presuppose that the deep dive was the only — or even 
principal — motivating factor for any actions taken following the study. Evaluation literature extensively 
discusses the “multiplicity of potential inputs to decisions” and the futility of attempting to attribute 
decisions made back to evaluation information, rather than simply recognising their contribution (Alkin 
and King, 2017). 


2. How the EIB-GDN programme deep dives were used 


This section summarises how deep dive information was used by the EIB and private-sector partners. 
It is based on stakeholder interviews, covering a sample of companies that were subject to deep dive 
research. All EIB members of the EIB-GDN Advisory Committee, four fund managers and six companies 
were interviewed. Three companies did not respond to the request for interview or declined to 
participate. As this chapter focuses on use, only those organisations that had experienced a full deep 
dive — and where at least preliminary results were available — are considered. This excludes companies 
in the 2019-2020 cohort. Names of individuals and partners have not been mentioned as all were 
offered anonymity. Some company examples have been included where information was already in 
the public domain or explicit consent was given. 


2.1. Instrumental use: Making use of deep dive findings for programming 


This use category covers programming in the broadest sense: making decisions about individual 
projects or company activities, as well as the use of results in reporting, including for promotional and 
fundraising purposes. 


At the company level, three distinct uses emerged. 


First, to plug data gaps. For some companies, this was the first structured data collection they had 
undertaken on impact. While others already collected some impact data, they recognised that they 
“always have gaps in understanding even when we try to measure impact.” 


Most companies already had access to user data — on who their customers were and socioeconomic 
profiles — even if this was not recognised as impact data per se, but collected for business marketing 
or strategic uses such as customer segmentation. Only one company felt it is “important for us to 
know our clients’ profile and it was something that we didn’t have any survey on before [the deep 
dive].” Most companies also had access to engagement and feedback data, but the deep dive helped 
to substantiate and scale this dataset. Beyond the resource constraints preventing companies from 
doing more customer-facing engagement, it was felt that using external researchers helped bring in 
independence, which increased the “weight” of the findings. Few companies already had data at the 
level of outcomes and attribution — covering higher-level changes and causality — and as such were 
areas where companies had been making large assumptions, which the deep dive helped to confirm 
or reject by generating evidence. 


6 At the time of research, nine companies had completed a deep dive and received the results of the exercise. The sample 
for this section therefore represents 66% of the population. However, given the small number of companies subject to deep 
dives overall, findings should not be generalised beyond the particular context of the EIB-GDN deep dives. 
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Second, the deep dives helped companies make or validate tactical decisions’. Data on the profiles 
and behaviours of key business stakeholders, such as customers, suppliers and communities, can help 
management make decisions about the way they operate their business. This includes deciding who 
they sell to, source from or employ, and learning how to maximise the value of these relationships. 
Several companies mentioned that the data helped finalise decisions already made concerning, for 
example, price points (the cost of the service) and marketing strategies (which product features to 
emphasise); the deep dive helped them move from intuition to more evidence-based decision-making 
(Box 2). 


Box 2: An example of tactical decision-making 


Two researchers worked with M-BIRR, a mobile money service provider in Ethiopia, to look at 
willingness to pay for an expansion of services to cover utility bill payments. An experimental 
technique — randomising hypothetical price offers — allowed the researchers to help M-BIRR 
understand the demand for this service. The main finding — that the willingness to pay is high and 
well above the cost — has strengthened M-BIRR’s interest in offering this service. 


Source: EIB-GDN (2019) 


Third, a smaller number of companies mentioned using the deep dives to inform strategic decisions. 
This included decisions about future business strategy, such as type of business model and which target 
markets to reach. Strategic decision-making also relates to how to strengthen a company’s social 
mission, including what partnerships to pursue (Box 3). 


Box 3: An example of strategic decision-making 


poa! internet provides wireless broadband in low-income communities in Kenya. As part of its 
business model, poa! provides unlimited free internet access to selected community institutions, 
including schools, in return for these facilities hosting the infrastructure used to provide its “street” 
internet service. The EIB-GDN study confirmed poa!’s view that access to unlimited data has 
changed the way students and teachers use the internet for teaching and learning. However, there 
was no evidence that provision of poa! internet had increased the probability that students received 
educational internet content at school outside of information and communications technology (ICT) 
classes, and students in schools with poa! access were not more likely to improve their ICT skills and 
knowledge. Although more work would be needed to identify the reasons behind these findings, 
poal’s management suspects that the full benefit of improved internet access can be felt only if 
appropriate hardware and capacity to manage the information technology system are in place. 
Responding to these findings, poa! is seeking ways to continue developing its schools service, 
potentially involving partnerships with non-governmental and other philanthropic organisations. 
Such partnerships could make it possible, in addition to providing free Wi-Fi, to cover some of the 
relevant infrastructure and capacity needs of the schools. 

Source: EIB-GDN (2019) 


7 Tactical decisions concern the detailed implementation of a strategy and are usually more focused on day-to-day 
operations. 
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From a company perspective, the deep dives uncovered little “new” knowledge, instead reinforcing 
what businesses knew - either intuitively or based on observation — but had lacked supporting 
evidence to demonstrate more empirically. Companies subject to deep dive research found the 
results and findings largely in line with what they were expecting. In the words of three interviewees, 
the studies “confirmed what we already knew,” were “findings that we hoped would come up,” and 
“packed up and quantified what we knew already.” Only one company did not anticipate the 
socioeconomic profile of its customers uncovered by the deep dive, but was “surprised in a positive 
way with the numbers.” 


It is unclear whether the deep dives largely confirmed existing assumptions because business models 
were already built on a solid commercial and impact case. A convincing argument can be made, for 
example, that entrepreneurs starting businesses and backed by experienced venture funds selected 
by the Impact Financing Envelope would already have had significant experience or a track record of 
business success on which to develop robust assumptions. There may have been bias inherent in the 
deep dives (covered in the next section) that led studies to focus on the most successful companies or 
on aspects of businesses that were already best known. 


Overall, there were few examples of businesses making concrete changes to the way they deliver 
value or measure impact as a result of the deep dive findings. According to companies involved in the 
final cycle of the projects, this was in part because major disruptions such as the COVID-19 pandemic 
meant that businesses had to shift to prioritising survival actions instead of those related to 
performance improvement. Another reason was that companies were, by their own admission, not 
systematic or consistent enough in putting in place an action plan to respond to the findings. 


There was one notable exception, where a company changed how it markets its product by shifting 
to more audiovisual methods. This company has seen 20% customer growth since the deep dive study 
shared its findings, although it is not yet possible to assess whether this is correlation or causation. 
Two companies said they would like to conduct a study again in a few years to see changes in impact, 
as a before/after study would provide a more solid foundation for strategic decision-making. However, 
both questioned whether they would have the financial or technical resources to be able to carry out 
such a study. One other company was planning to collect some of the same feedback data collected 
by the deep dive themselves so they could track changes and improvements over time. 


Two companies reported using the findings of the deep dive to engage with local stakeholders such 
as legislators and policymakers. They used the studies to convince these stakeholders that their 
operations were socially acceptable and legitimate; in other words, to cement their social licence to 
operate (Raufflet et al., 2013). For these companies, the value of the deep dive approach was when it 
was geared towards an external audience to help “influence public policy and decisions.” Here, deep 
dive learning took place at an industry level and companies felt “it would [have been] helpful to have 
other industry participants involved [...] not just the narrow focus on a company.” This higher level of 
instrumental use focuses on building a supportive business environment for companies introducing 
new products and services in the market. 


A small number of companies noted disappointment that issues they had thought were critical to 
making operational and strategic decisions were not covered in the report, or were covered in 
insufficient depth, mainly due to the study attempting to cover too many topics. 
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In terms of how the companies communicated the deep dive information within their own 
organisations, none shared the full deep dive report internally (within the company) or externally 
(to shareholders and stakeholders). Rather, the deep dive summary or the presentation made by 
researchers was circulated amongst the management team, and in one instance to the company’s 
board. Sharing summaries rather than the report was necessary, according to one respondent, so that 
findings could be understood by “anyone who was not a statistician.” One company put together a 
bespoke one-page slide deck on findings, with the help of the researchers. The company reported 
doing “a presentation to the entire office [...] [and then] extracting one slide and working [it] into 
marketing team [materials],” which was used for external promotion. Two companies integrated 
selected deep dive data points and findings — so-called “snippets” — into their investor pitch decks. 


Investment funds echoed that, as expected due to the early-stage nature of many of the companies 
they invest in, entrepreneurs would have many “hunches” about impact but lack data and evidence. 
As well as the intrinsic value for company decision-making, the deep dives were primarily useful for 
investors in confirming an investee’s impact thesis/hypotheses. As one fund manager noted: 
“Sometimes when investing in companies we make estimates about impact — [we are] very happy to 
have those challenged or confirmed through research.” Deep dives help reconfirm the rationale of the 
investment from a social impact perspective. There was one example of a fund using the results of the 
deep dive study in a brainstorming session about their new theory of change, thus feeding into 
portfolio-level decision-making. A number of other funds took highlights from reports and used them 
for annual sustainability and/or impact reporting, saying this allowed them to go into “more depth 
than normal” on impact. 


From the perspective of instrumental use, deep dive information helped reinforce the “feedback 
loop” between the EIB and the investment funds and companies receiving direct financing — “project 
promoters” in EIB terminology. Respondents felt that sharing deep dive results strengthened dialogue 
between the EIB, project promoters and investee companies on impact. No specific examples of using 
deep dives for decision-making were given, mainly due to the type of financing provided: three of the 
four Impact Financing Envelope instruments are indirect financing, so the EIB does not always have a 
direct line of sight into company operations, meaning that company-level decision-making is largely 
the job of a fund manager or financial intermediary. 


There was a strong promotional purpose for the EIB in conducting the deep dives. They have been 
used to illustrate the Impact Financing Envelope’s impact based on data and evidence, in the form of 
blogs, working papers and conference presentations®. This helped the EIB show what is achieved in 
terms of project-level outcomes, which goes beyond outputs — largely the focus of the results 
framework — in a way that is backed up by “academic credibility.” The audience for these pieces is 
mainly external, supporting the wider EIB communications work and demonstrating that the Bank is 
not just about lending money, but can lead to concrete, credible and externally verified results. As one 
respondent noted, “part of the [deep dive] story was trying to demonstrate the impact of [the] EIB.” 


The findings of the deep dive studies were presented to various governance bodies of the institution, 
including the committee of the ACP Investment Facility, made up of the Member States who oversee 
the Impact Financing Envelope. Feedback from these “shareholders” was noted as extremely positive, 
according to an EIB stakeholder, especially since members welcomed the opportunity to see more 


8 All published outputs of the programme can be retrieved at: The EIB-GDN Programme in Applied Development Finance. 
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granular, project-specific results of EIB financing. The deep dives therefore fulfilled an “institutional 
positioning” function to help strengthen the EIB’s reputation as a “proper development finance 
institution” that can successfully operate outside of the European Union. Internally, studies were 
circulated to individual loan officers and technical colleagues such as economists. An international 
conference, co-organised by the GDN and the EIB in partnership with the Campbell Collaboration, was 
held on 2 to 4 December 2020, titled “Assessing the Development Impacts of Private Sector Impact 
Investments.” It reflected on various lessons from the deep dive experience. Two webinars were held 
on 24 and 26 March 2021 to discuss “Assessing Impact Investments,” co-hosted by the EIB and the 
GDN. 


One limitation for instrumental use within the EIB is that the deep dives focused on impact on end 
beneficiaries such as micro, small and medium enterprises, customers and communities. Here, the 
impact is being created by companies and is facilitated by investment. Much of the financing under 
the Impact Financing Envelope is indirect (via intermediaries) — and deep dives do not directly cover 
the role of EIB financing and support, aside from any assumed pooled capital effects. This “investment 
impact” — a particular investor’s financial contribution to the social value created by an enterprise 
(Brest and Born, 2013), as summarised in Chapter 1 —is an important part of the EIB’s contribution but 
was not covered by the deep dives. This underlines the importance of aligning deep dive findings with 
those emerging from more formal evaluation exercises, which often take the fund level as the unit of 
analysis and ask questions about investment impact and additionality so that “impact studies” can be 
designed to complement the findings emerging from an organisation’s own evaluation work, which is 
often commissioned or carried out by independent evaluation departments’. 


2.2. Conceptual use: Enhancing knowledge about the type of intervention under study 


As expected, this user pathway was more relevant to financiers — the EIB and intermediary funds — 
than individual companies that are more focused on their specific business model than generalisable 
learning. 


The deep dives were never intended to have a thematic focus. While discussions at early EIB Advisory 
Committee meetings grouped shortlisted projects under major themes, these covered different types 
of Impact Financing Envelope instrument, such as risk sharing, social impact funds and direct financing, 
rather than specific thematic areas. In part this is because the EIB Evaluations Operations function 
already has an established process for thematic evaluations, including for operations outside of the 
European Union, covering topics such as financing water and sanitation projects, and climate action”®. 


One of the benefits EIB stakeholders perceived about the deep dives was the ability to complement 
thematic evaluations, which by design provide “wide but shallow” strategic insights, by focusing on 
individual projects to allow for “narrow but deep” operational insights. A granular focus was therefore 
“an advantage [in] that it could shed light on project-specific results.” 


A lack of a thematic focus was also partly because of feasibility. The deep dives were drawn from a 
relatively small pool of projects — those funded by the Impact Financing Envelope — so there was not a 


° For example, the independent evaluation of the 250m Africa-focused impact fund-of-fund FISEA, the French Development 


Agency: https://www.stewardredqueen.com/what-we-do/fisea/. 
10 https://www.eib.org/en/publications/evaluation-of-eib-financing-of-water-and-sanitation-projects-outside-the- 
european-union and https://www.eib.org/attachments/ev/ev_activity report_2019 en.pdf 
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large pipeline of new projects each year to choose from. The Impact Financing Envelope was also a 
new initiative and the deep dives started alongside it, meaning that the initial pipeline of projects was 
small. Companies were selected not by criteria but purposively, based on the project promoters and 
companies that were available and willing to participate. Sometimes it was a significant challenge to 
find enough projects “willing and able” to participate in a deep dive study, so no real choice between 
companies was possible. However, an attempt was made to achieve a coverage of different Impact 
Financing Envelope instruments and geographies, particularly the Caribbean and anglophone and 
francophone Africa. The selection of projects was constrained by the location of the researchers and 
their backgrounds and expertise. The researchers were recruited and then assigned to companies, 
rather than being hired on the basis of their experience and skills match to carry out a specific study. 


However, one EIB stakeholder noted that “a thematic focus would have helped [the deep dives] be a 
more useful exercise,” even if it was around crude groupings such as off-grid solar, financial technology 
and digitalisation, rather than specific thematic research questions. They noted a thematic focus would 
have helped generate more internal knowledge to feed deep dive learning into future operational 
decisions, such as by informing due diligence for other deals in similar thematic areas. 


There are questions as to the generalisability of knowledge generated by deep dives. Even if a strong 
intentionality for conceptual use was built into programme design, it was unlikely to go much beyond 
broad categories such as “microfinance.” This echoes the experience of other development finance 
investors such as CDC, which found limitations in turning insights from individual deep dives — however 
useful for the company in question — into replicable learning for other investees and investors (Castro 
and Ripley, 2019). 


One fund manager interviewee expressed a strong desire to extract generalisable learning from the 
deep dive studies and thought that the programme had missed an opportunity to share learning across 
studies to help private-sector partners see results from those they were not directly involved in. It was 
suggested that a smart grouping of themes could be done against topics likely to attract maximum 
interest by private-sector partners, such as by business model typology. For example, as that investor 
tends to back business-to-business (B2B) companies — business models that focus on selling services 
to other businesses — they would be interested in what other B2B-focused deep dives were learning 
about how to measure and maximise impact. 


There is currently little evidence of conceptual use, but it may happen in future. With 15 studies soon 
to be completed, the deep dives can be considered a “body of knowledge” that can be tapped into and 
built on over time. Conceptual use can take place after the fact, in that it is possible to code and 
synthesise thematic knowledge even after a study has been completed, should there be a critical mass 
of studies from which to extract learning. 


2.3. Symbolic use: Fulfilling a requirement to do a deep dive or to show support for an 
intervention area 


The act of completing a deep dive study partly fulfils the “use” requirement for the EIB, as it shows 
that rigorous methodologies are suitable for applying to private-sector investments. One EIB 
stakeholder explained the whole programme could be thought of as a “proof of concept” establishing 
that deep dives can be carried out. Another interviewee highlighted the importance of “demonstrating 
[...] [and] showing to colleagues that undertaking these studies is possible — providing some conditions 
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are met.” The process of doing a deep dive has important value that goes beyond the specific technical 
learning, which is covered in the next category on capacity building. 


Funding deep dive studies has a symbolic use in that, according to another respondent, it “shows 
the EIB is serious about impact.” This underlined the importance of bringing in renowned experts in 
the field of impact assessment and development evaluation. It was felt to provide a signalling effect, 
showing that if “people with international reputations from respected organisations” were involved, 


|’ 


that this was a “serious and professional” exercise. Coupled with this was the credibility from 


collaboration with academia: “trying to use more rigorous [methods] and most recent tools.” 


There was an element of symbolic use in private-sector partners, as a small number of respondents 
expressed that they agreed to participate in a deep dive because they felt obliged to, given their 
investors were supporting it. For these companies, the first objective was to “do no harm” by not 
allowing the study to “get in the way” of running the company or, in the words of one fund manager, 
not to “distract” the company. The objective was thus to minimise potential burdens. 


2.4. Capacity building use: Strengthening the management of an organisation or project 


The type of use covered here focuses on organisations rather than on individuals, which was covered 
in Chapter 3 on the value created by training a set of talented young researchers from the sub-Saharan 
African, Caribbean and Pacific regions. A number of respondents questioned the efficacy and 
effectiveness of focusing capacity development at the level of researchers. Rather, they thought, 
efforts should have been focused at the institutional level where outcomes could have been scaled 
and stood a better chance of being sustained. One respondent believed the best “approach would 
[have been] more about building institutions rather than targeting individuals.” 


From an EIB perspective, there was considerable process knowledge generated through the deep 
dives, which contributed to wider organisational learning. The EIB’s current approach to evaluation 
is largely theory-based contribution analysis, with little or no use of counterfactual methods. The 
intention in this programme was to apply impact evaluation techniques and see if they could 


|’ 


complement other tools such as macromodelling or “traditional” evaluation using the Organisation for 
Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) criteria 
(OECD, 2002). This was an opportunity for the EIB to learn whether the deep dives were a useful 
approach and how to go about doing them. EIB respondents felt that deep dives “generated replicable 
process knowledge [...] on very practical aspects of impact evaluation.” This included on issues such as 
how to clarify an intervention logic, how to produce a solid baseline, and how to demonstrate a 
credible causal relationship between an investment and observed outcomes. “These are lessons of use 


for all projects,” noted one EIB stakeholder. 


EIB respondents felt that deep dives had contributed to spreading a culture of evaluation across the 
organisation, and that the programme had successfully made the case for why it was important to 
consider rigorous methods. However, respondents were realistic that deep dives were one small 
contribution in a large institution, and it was a bit early to see the difference. It was also felt that, 
currently, knowledge rested more at individual rather than institutional level. Process knowledge took 
place through more informal mechanisms, such as by commenting on studies, rather than any formal 
or codified capacity building targeted at the EIB. 
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There was no evidence that deep dives contributed to strengthening company management from a 
content or process knowledge perspective. There was little interest from private-sector partners to 
learn how to deploy more rigorous impact measurement techniques, mainly because they lack the 
budget, expertise and incentives to, in the words of one company, “invest in academic rigour.” The 
companies did make use of deep dives — there was considerable instrumental use — but had low 
interest in developing their own internal capacities to collect impact data, beyond limited topics such 
as user, engagement and feedback data that do not require the application of academic research 
techniques. 


3. Factors influencing deep dive use 


This section identifies key factors influencing deep dive use, focusing on the quality of use. The 
emphasis is on “appropriate use rather than just greater use” (Alkin and King, 2017). The aim is to link 
the question of “how” deep dives were used to the “why” reasons for participating (as set out in the 
conceptual framework in Section 1 and Figure 1), with reference to the operating and organisational 
context of deep dive user groups. 


Evaluation literature categorises use factors into the following four groupings (Alkin and King, 2017), 
which can be applied for the purposes of understanding factors influencing deep dive use: (1) user 
factors; (2) evaluator factors; (3) evaluation factors; and (4) organisational factors. 


3.1. User factors 


The work of evaluators such as Michael Quinn Patton identified the “personal factor” as an 
“important element in determining whether people used evaluations” (Alkin and King, 2017). This 
topic covers how user “attitude[s] towards evaluation, both in general and specific to the current 
evaluation” can influence use. These attitudes, in turn, are thought to derive from how meaningfully 
users are involved in the evaluation; user predispositions about evaluation, based on past experiences 
(both positive and negative); and innate user interest in the topic and process of evaluation. Each of 
these are examined from the perspective of deep dives. 


The involvement of user groups 


Patton (2008) argues that “intended users are more likely to use evaluations if they understand and 
feel ownership of the evaluation process and findings [and that] they are more likely to understand 
and feel ownership if they've been actively involved.” 


The deep dives were participatory and involved both EIB and private-sector partners. The extent to 
which user groups were meaningfully involved varied across studies and within organisations. From an 
EIB perspective, a small number of staff were involved in actively supporting the framing of research 
questions, providing comments on research plans and instruments, and undertaking significant editing 
of final reports. However, broader EIB organisational units were only marginally involved, at most in 
commenting on draft deep dive reports". 


11 Greene (1988) highlights the importance of stakeholder involvement, distinguishing among the “very involved person,” 
“the somewhat or sometimes involved person,” and the “marginally involved person.” 
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In general, fund managers were hands-off, in part because they did not want to take ownership of the 
process away from investee companies. However, this may have contributed to funds playing a less 
proactive role in the instrumental use of deep dive information or understanding the depth of insights 
being generated. 


The investee companies subject to deep dive research were all involved, as would be expected in 
any research exercise focusing on their business performance and impact. In most instances this was 


é 


“occasional” involvement, coming in at strategic points such as the selection of the research focus to 
ensure the “study covered subjects that really mattered” (company interview). In some instances, 
companies were deeply involved, especially when they designated a counterpart to work closely with, 
or even alongside, the researchers. This was often a staff member in charge of in-house analytics or 
market intelligence, such as a data science lead. Indeed, the strongest examples of use — where 
feedback from private-sector partners indicated the deep dives were most useful — came from studies 


where a knowledgeable counterpart was appointed. 


Views on impact measurement 


Impact Financing Envelope investments, like impact investments more widely, flow into businesses 
that create impact in different ways. Some companies are more mission-driven with an explicit social 
purpose, such as a social enterprise whose motivation is the double or triple bottom line of financial, 
social and environmental performance (so-called “investing for impact” strategies) (West, 2019). Other 
companies may be more mainstream commercial operations, but still expect to create social impact 
due to the nature of the products and services they sell (so-called “investing with impact”). The 
enthusiasm for impact data varies depending on, in the words of one respondent, “how much social 
impact is tied to the commercial engine.” Mission-driven enterprises will see impact as fully aligned 
with the commercial engine, while others are “pretty much for a profit [...] [and] tack a little bit of 
impact on the side” (company interview). For the former, deep dive information may be “need to 
know,” while for the latter deep dives generate “nice to know” information, unless the impact data 
can be commercialised for business value purposes. 


A small minority of more commercially oriented private-sector partners cited negative experience 
with other impact measurement exercises in the past as the reason why they were initially sceptical 
of the deep dives. In the words of one investee respondent, the “excessive weight [being placed by 
the impact investing industry] on impact measurement [...] is a gigantic waste of time.” These private- 
sector partners did also end up deriving instrumental use of the deep dive, using findings to inform 
marketing and business decisions. However, in their mind “impact” data was synonymous with “top- 
down” reporting metrics set by investors that do not align with “on the ground” business data needs. 


Business model context 


Companies subject to deep dive research spanned the business life cycle, from more “early-stage” 
start-ups still refining their business offer, to more “growth” stage companies seeking to expand their 
reach and penetrate new markets. The type of products and services also varied from those with long 
histories (for example, microfinance) to others developing more innovative products and services (for 
example, solar-powered micro-utilities). 
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The differing company contexts shaped demand for different types of impact data, and how useful 
these companies found the deep dive. Studies were always most useful when they were future- 
looking — that is, aligned to the company growth strategy not just their historical impact — and there 
was a Clear distinction between companies that emphasised the need for speed versus those 
prioritising the importance of rigour. 


Two separate groups of needs can be identified for private-sector partners: those that wanted more 
“academic” evidence and those that wanted more “actionable” insights. 


The “academic” group 


These companies primarily sought to use deep dive information for marketing and fundraising 
activities, or to engage policymakers. These companies had often built their business model around 
anecdotal or secondary data; for example, where research conducted by other companies and in other 
countries had shown the value of using mobile wallets, but empirical data did not yet exist in the 
company’s own context. 


For these companies, achieving the highest level of rigour possible in the study was paramount. The 
deep dive, in one company’s words, had to be “really good to have some potential for fundraising for 
[the] company in future.” There was a clear value added in the approach taken to the deep dives, as 
the most rigorous methods were never “something that would be able to justify spending time and 
budget doing ourselves.” 


The independent nature of the study was important, especially when used to validate aspects of the 
business model for an external audience. As one respondent noted, there is a significant difference in 
being able to confidently state in marketing materials that “this is what the customers/the market 
stands to gain from working with company X (and it’s not company X saying that).” 


Even though these companies recognised that the research process needs to be thorough, many felt 
the exercises were too open-ended and studies took too long to finalise. Two years on, one company 
said it “didn’t feel like [the deep dive] ever finished.” As a result, some private-sector partners were 
unsure when the exercise was completed, and when they could start to use the results. 


The “actionable” group 


Venture-backed companies sell more innovative products and services, and have less fixed business 
models. In the jargon of the start-up world, these companies are agile (continually iterating 
products/services to better meet user needs) and pivot often (shift their business strategies). As one 
respondent noted when reflecting on methods for measuring impact, “academic models [that assume 
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a fixed treatment] are not so usefu 


These companies have various types of information needs, but tend to value user, engagement and 
feedback data in order to make rapid decisions about customer segmentation, satisfaction and 
willingness to pay/use more than outcome and attribution data. While wanting data to have a degree 
of rigour — in the sense of being thorough and carefully collected (ILO, 2014) — the entrepreneurs and 
decision-makers guiding these companies prioritised the speed of data. As one said: “Quicker and 
faster is better than deep academic rigour.” For them, deep dive findings do not have to be perfect, 
but do have to be timely. 
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Because start-ups move quickly, there is a danger if deep dives are not timely that insights arrive too 
late: the business might have already had to make decisions, so there is no chance to implement or 
act on findings, or worse still the companies may have pivoted and the subject of the deep dive study 
is no longer relevant. As one EIB stakeholder reflected, “in terms of companies evolving quite rapidly, 
[our] approach was too slow. It wasn’t delivering real-time info that these early-stage companies 
wanted.” 


3.2. Evaluator factors 


The profile and performance of the evaluators has long been emphasised in the literature on 
evaluation use. Relevant factors identified include the dedication and commitment of the evaluator, 
interpersonal skills (“developing rapport and a good working relationship with those users”), political 
sensitivity and credibility in terms of experience, technical expertise and contextual knowledge (Alkin 
and King, 2017). 


One objective of the EIB-GDN Programme in Applied Development Finance was to provide a 
“training and mentorship programme for local researchers” (EIB, 2021). By definition, the researchers 
leading the deep dives were relatively inexperienced, even though their technical expertise was 
assured during the selection process and they had the support of a panel of expert advisors. Few 
researchers had previously been exposed to business operations, and few of the expert advisors 
specialised in impact investing or private-sector development. 


All user groups interviewed for this volume provided positive feedback on the individual researchers 
they engaged with as being professional, and there was a high degree of satisfaction on the part of 
the private-sector partners. There was a question as to how local the researchers were, given few 
were based in or from the project countries. In one example where the deep dive findings were used 
extensively by the company, the lead researcher was based in the same city and was teaching at the 
same university from which a number of company staff had graduated. Another company based in 
East Africa, however, noted significant cultural clashes with their West African researcher, who had 
different expectations about how fast people and processes move in-country, which initially “created 
tensions” (company interview). 


Adapting academic techniques to a private enterprise context is complex. It requires situational 
leadership — the ability to change language and style to meet the needs of different user groups. The 
researchers needed to engage in deep technical discussions with expert advisors, but also 
communicate sometimes complex ideas with companies in a business-friendly way. There was a sense, 
according to one respondent, that this “was two different worlds coming together.” Sometimes 
researchers and businesses “were talking two different languages [...] and things got lost in translation” 
(company interview). This led to gaps between what the company was thinking and what researchers 
were doing. This led one company to conclude the “for-good lingo and for-profit lingo are mutually 
exclusive.” 


In the words of one investee respondent, for the vast majority of researchers this was the first time 
they were doing a survey in the business world. Undertaking a survey of paying customers engaged 
in a chain of commercial transactions is different from surveying “beneficiaries” in an international 
development project accessing a grant or a free service. Different power dynamics introduce different 
biases, and research has shown the potential for the research process itself and the type of questions 
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being asked to influence customer behaviour (Brooks and John, 2018), having a potential impact on a 
business’s ability to attract and retain customers. As one company mentioned, researchers need to be 
more mindful that “these are customers not participants in a project,” and so the “social capital of the 
company is expended” in any contact with them. The researchers were largely learning by doing, which 
is pedagogically effective, but less so in terms of generating the highest quality insights. 


Structurally, some EIB stakeholders felt that researchers were pulled between meeting company 
needs and the researcher’s own interests (for example, as PhD candidates), and the expert advisors 
(who often had a more rigorous mindset). Private-sector respondents suggested several 
improvements to the approach, including researchers obtaining a more solid grounding in how 
businesses work and ensuring researchers did earlier field visits’?. A number of companies believed 
that too much time was spent planning the deep dive from afar, but researchers were hit with field 
reality when they came to the country and had to correct course quickly. Companies also cited 
communication as a challenge when researchers were working remotely (even prior to the COVID-19 
travel restrictions), particularly during the early planning stages and later data analysis and report 
writing stages. 


3.3. Evaluation factors 


This set of factors distinguishes between the role of the individual evaluator and the process of 
evaluation and the activities within it (Alkin and King 2017). From a use perspective, “while technical 
excellence and necessary rigour are important, what is most relevant is the appropriateness of the 
methods employed and their credibility with potential users” (ibid.). 


These factors can be grouped under the heading of relevance (Yarbrough et al., 2010). Four aspects 
are examined with reference to the relevance of deep dives: the overall approach, choice of 
methodology, timing and presentation. 


Deep dive approach 


The deep dive programme was funded by the EIB to deepen understanding of the results and impact 
of operations under the Impact Financing Envelope. In the hierarchy of user needs, the intention was 
primarily to benefit the EIB and researchers through capacity building. The fact that private-sector 
partners (companies and funds) could also derive benefit was largely of secondary consideration. As 
one EIB member noted “the user-centric expectation was not really there [at the beginning] — or at 
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least centra 


To attract company participation, every deep dive was pitched to companies as also being needs- 
led. Over time the deep dives did evolve more of a “bottom-up” user focus to complement the top- 
down investor perspective. However, as company needs were always one of many (sometimes 
competing) objectives, private-sector partners did not always see the deep dives as tailored to their 
interests, or fully aligned with information needs. As one company questioned: “Who was the real 
audience?” The EIB was open that the primary private-sector audience was “probably not really 


12 Companies also felt that researchers would have been more effective if they could have “teamed up” with someone with 
knowledge of the reality of business operations, and if there was a better matching of researcher interests to company data 
needs. 
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companies themselves, but maybe the wider community of [...] investors and fund managers in SMEs 
in developing countries.” 


A fund manager respondent noted that “starting from the field [reality] is a fundamental driver of 
encouraging, adopting, demand and use.” Instead of a centralised approach that sought to structure 
a programme and then get private sector “buy-in,” they suggest a more decentralised model to “ask 
what problems companies [have] and [then] put together [a] programme,” more akin to a flexible 
technical assistance facility. 


Methodology 


EIB stakeholders highlighted the importance of the high-level academic expertise that the GDN and 
the expert advisors brought to the programme in ensuring that rigorous methods were used and in 
giving the studies a “quality stamp.” The EIB believed that the credibility gained through this approach 
would lead to increased use of the studies and findings. 


However, EIB stakeholders also noted that the methodology used in the individual studies had to be 
selected pragmatically to fit with the various constraints discussed in this volume, such as timing 
and funding. The EIB saw the aim for each study as arriving at an “appropriate degree” of rigour. Given 
the programme constraints, the objective was to do “the best out of what was possible,” in some cases 
where it was a challenge to even start the research, “even if [the study was] not perfect; it is better to 
have something than nothing.” For example, it was not always feasible to estimate the counterfactual 
using a control group, so alternative methods such as comparing different customer segments 
(between newer and older users) or trend analysis were explored. In other cases, budget limitations 
and the need to minimise the burden on private-sector partners made it difficult to implement more 
than one round of data collection. This limited the depth of the conclusions that could be drawn from 
the study. EIB staff suggested that acknowledging the practical constraints and adopting a narrower, 
more realistic focus from the start of the research process could have helped, in some cases, to make 
the final studies more useful: many studies “tried to do [...] [and] to answer too much” (EIB interview). 


The vast majority of private-sector partners greatly appreciated the participatory nature of the deep 
dive methodologies. In the words of one partner, it allowed companies to get hands on to ensure that 
studies “covered subjects that really mattered.” The rigorous element of the research was, as 
mentioned above, something that companies lacked capacity to carry out themselves, so such an 
“academic” lens was seen as a distinct value added. 


However, a small number of partners expressed concerns about methodology, particularly 
frustration with the difficulty of understanding the methods chosen, and that the “overly 
econometric approach [...] [was] hard to understand.” Some also expressed concerns about the 
reliability of data on topics like increased income, which they knew were difficult to measure, or about 
the appropriateness of inferring causality based on the findings — a concern that was echoed by some 
EIB stakeholders. There were also isolated cases in which the studies delivered did not fit the standards 
the private-sector partners had expected in terms of quality of the analysis or where partners believed 
that specific methodological concerns affected the usefulness of the work. Issues raised included the 
potential for respondent biases to arise, particularly when the researchers introduced themselves as 
surveying on behalf of the company, the representativeness of the samples and the framing of some 
survey questions. 
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Timin 


Evaluation literature states that “information [...] must be timely, or it may be of no use to potential 
stakeholders” (Alkin and King, 2017). The time it took to complete the deep dives was an issue for all 
companies, particularly the earlier-stage enterprises. In part this was an issue of managing 
expectations. As noted by one respondent, the “initial timeline was way too optimistic.” For other 
companies it “wasn’t clear up front how long it would take” and “in our mind [we were] thinking about 
something shorter.” 


Presentation 


Literature on evaluation use is unequivocal about the importance of “communication quality, 
including evaluation reporting” as a factor affecting use (Alkin and King, 2017). Evaluating findings 
“must be in a form that users can understand” (ibid.). 


Reflecting on the overall programme, an EIB stakeholder believed the standard for what could be 
considered good quality should reference both the rigour of the study itself and its readability. For 
this EIB user, in some studies it was difficult to understand whether what was being presented was a 
positive or negative finding, and that the “simplicity of messaging [was] often not there.” Interesting 
insights and important findings were therefore being obscured as the deep dive was not, in the words 
of one company, “properly presented,” making the study hard to share. 


This was echoed by some private-sector partners, who critiqued a “highly theoretical research paper” 
and questioned whether it was “written for [a] business audience or [...] peer reviewers.” One company 
stated they were “not a scientist” and so did not understand the full report; another stated that the 
study was “not the most exciting piece of work to chew through” and as a result, “the format buries 
any interesting findings.” 


Over time, shorter summaries and slides presenting the key insights from the deep dives were 
prepared. This goes some way to meeting key criteria to make reports readable, understandable and 
useful, as set out by McDavid et al. (2013): 


= Rely on visual representations of findings and conclusions where possible. 

= Use clear, simple language in the report. 

= Use more headings and subheadings, rather than fewer, in the report. 

= Prepare a clear, concise executive summary. 

= Be prepared to edit or even seek professional assistance to edit the penultimate draft of the 
report before finalising it. 


|“ 


poor product quality” (Mayne, 
2014), minimising the chance the deep dive was useful. Deep dives were frequently “not timely” (did 


In conclusion, a small number of evaluation factors led to occasiona 


not produce findings when they were needed for decision-making) and “not clear” (findings not well 
communicated, with no communications strategy and no recognition of different stakeholders’ 
interests). In a few isolated cases, users believe that a deep dive was “not credible,” in that the 
evaluation findings, conclusions and recommendations were not seen as high-quality enough to be 
used in decision-making, due to weak data and/or analysis. 
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3.4. Organisational factors 


This wider set of factors covers both the organisational and social contexts underpinning deep dives. 
Evaluation literature shows that “the history, nature and value of the organisation in question also 
clearly will have an impact on the use” (Mayne, 2014). This topic, however, is too broad to cover in any 
detail beyond some broad observations. 


First, the nature of the organisation in which the evaluation is being conducted has a substantial 
impact on the successful achievement of deep dive use. This includes the characteristics of the 
programme in question — in the deep dive case, the Impact Financing Envelope — as well as the larger 
entity that encompasses that programme (Richard et al., 2009). The EIB is a large international financial 
institution, and any contribution the deep dives make to cultural change has to be understood in this 
wider organisational context. An EIB respondent noted that deep dives were aimed at helping 
colleagues to “better describe and learn from not just where [they] put their money but what the 
results are.” This view was echoed by a private-sector respondent who saw the value in the deep dives 
helping to give the EIB concrete insights into the effects of the products and services of the businesses 
receiving funding, allowing it to see whether, in practice, the impacts matched those that the Bank had 
expected to see based on theory. Some of the private-sector respondents held the view that the “EIB 
is a development and an investment institution — not an academic one. They shouldn’t produce books 
but [learn how to achieve] better outcomes on the ground.” 


The nature of the organisation on which the evaluation is being conducted also has a substantial 
impact on use. Corporate cultures are diverse, but as shown earlier, there were a set of private-sector 
partners deploying a start-up mentality that emphasised rapid, quick and dirty data, which is the 
antithesis of the slow and careful academic research techniques favoured by deep dives”. In terms of 
instrumental use, the most significant set of organisational factors shaping decision-making were likely 
to be sources of information other than the specific deep dives, but these are outside the scope of this 
volume (Patton, 2012). 


4. Conclusion 


“At its simplest, judgments about an evaluation’s utility are made based on the extent to which 
programme stakeholders find evaluation processes and products valuable in meeting their needs” 
(Yarbrough et al., 2010). 


Rockefeller Philanthropy Advisors, part of the foundation that first coined the term “impact 
investing,” recently reissued its seminal handbook with a chapter on impact measurement. Good 
measurement, it writes, should help organisations “manage forward” to improve impact over time, 
rather than just look back at what impact has occurred (Rockefeller Philanthropy Advisers, 2020). This 
way, impact measurement and management can be viewed not as a necessary cost, but rather as a 
practice that can inform many aspects of business performance. 


The recent focus on measuring impact to drive better decision-making is analogous to the utilisation- 
focused evaluation approach developed by Michael Quinn Patton over a decade ago. The core 


13 An exception is where the EIB-GDN deep dives partners with the DFID Impact Programme deep dives. The Impact 
Programme supported less rigorous but more real-time telephone surveys on customers over a six-week period, while the 
EIB-GDN deep dive carried out a quasi-experimental study over the course of one year. 
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principle is that any evaluative exercise should be judged on its usefulness to its intended users. “If the 
evaluation information collected does not meet the perceived need of users, then, not surprisingly, 
they are not likely to use it,” however technically robust the exercise (Alkin and King, 2017). 


Historically, many academics believed that “the quality of an evaluation rests on its methodological 
defensibility, and that short term, specific uses are not that important” (Fitzpatrick, 2002). Most 
evaluators now recognise that use is central to the reasons for doing evaluations. However, a tension 
remains between use and methodological defensibility; that is, achieving the highest levels of rigour 
and user-centric needs such as speed and practicality. This tension was felt throughout the deep dives; 
it shaped the issue of whether, and to what extent, the studies were useful. 


The EIB-GDN deep dives focus on methods as a “systematic exercise in applying academic research 
techniques to understand the impact of private-sector investments,” according to one EIB 
interviewee. From an instrumental perspective of utilisation, rigour matters only up to a point to 
ensure the data and insights are accurate enough as the basis for decision-making. For specific user 
groups like the companies being studied, academic rigour does not need to be maximised, but should 
be employed to a level appropriate for the intended use of the information. A company will not require 
a control group, for example, if its needs are simply to find out what existing customers think of its 
product. 


Multiple use pathways existed within the EIB and among the EIB’s private-sector partners. While the 
EIB derived broad conceptual and symbolic use from the deep dives, most private-sector partners 
found instrumental use in terms of validating business decisions, marketing and strengthening 
stakeholder engagement. These differences meant that different user groups sometimes had opposite 
experiences of the same study; for example, the deep dive cited as most interesting and useful to EIB 
stakeholders was the least useful from a company perspective”. This reflected the structured feedback 
given by private-sector partners, which was split between those who said the deep dive they 
experienced was moderately useful (40%), and those who said it was very (50%) or extremely (10%) 
useful”. For the EIB, a greater thematic focus and a broader pool of projects to choose from could 
have strengthened the use case. For the private-sector counterparts, on the other hand, particularly 
for earlier-stage companies with rapid product cycles, timelier findings that were easier to act on 
would have boosted use. 


Given the mixed use made of these studies, decisions about increasing the usefulness of the deep 
dives should be made based on clear decisions about how the studies might be used, and which of 
these uses should receive priority. The core challenge, as noted by an EIB interviewee, is to “adjust 
[the level of] rigour to show really useful results depending on the audience.” Striking the balance is 
hard, especially when there are competing user priorities. Placing too strong a focus on methods and 
seeking to apply “the highest academic standards and rigour” is unnecessary if user needs are for 
simpler and more actionable insights. Yet a too utilitarian view of impact studies that prioritises use 
over methods risks turning deep dives into pure market research. As one fund manager felt, this would 
bring into question the additionality and value for money of the deep dive approach, as for all data 


14 As measured by a standard question asked to all private sector participants on the usefulness of the deep dive. 

15 The Net Promoter Score (NPS) is a standard score that measures satisfaction and likelihood to recommend a product, 
service or approach to peers. Private sector partners gave the deep dive experience a relatively low NPS of 10: with 20% 
classified as detractors (not satisfied), 50% as passives (undecided whether they valued the exercise), and 30% as 
promoters (who value the deep dive and are likely to speak positively about it to others). 
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points aside from outcome and attribution, “companies can make decisions themselves [...] or if they 
need data they can hire in a local provider much more quickly.” 


The objective of increasing the usefulness of the studies also has to be balanced with other 
objectives, such as those connected to capacity building, and with the varying perspectives and 
experiences of researchers and their expert advisors. One stakeholder wondered whether it is too 
much to expect significant deep dive use, and that the programme may have been “over-ambitious.” 
As another respondent noted, it is “never good to [try to] achieve two objectives at once.” 


Overall, the deep dives were, in the words of one EIB respondent, a “fantastic exercise” in learning 
about impacts and how best to measure them. The experience of implementing and improving three 
cycles of deep dives has provided a clear framework for use, and a typology of different uses based on 
their real-world application. Furthermore, the experience has taught staff and management a number 
of practical, methodological and conceptual lessons about how to implement this kind of research. As 
one EIB interviewee noted, this process knowledge was as important as the findings of the studies 
themselves. 


Deepening the extent to which future cycles of deep dive information is used, particularly for 
private-sector partners, would likely require an adjusted approach. This would start by identifying 
clear and specific user needs and breaking the broad concept of use down into its component parts: 
why undertake a study; who are the primary users; what information do they need; and how will they 
use findings to prove or improve their impact. Deep dives would then be designed to “ensure that 
these intended specific uses [...] by the primary intended users guide all other decisions that are made 
about the [...] process” (Better Evaluation, 2012). 
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billion in development finance. She started similar groups in the Inter-American Development Bank 
and the Africa region of the World Bank. An economist by training, she has been an advisor to 30 
multilateral and bilateral development agencies worldwide. She was an expert advisor in the EIB-GDN 
programme. 


Debora Revoltella co-authored the Introduction. She is director of the Economics Department of the 
EIB, which is in charge of the sovereign and financial sector rating models for the EIB as well as 
providing financial sector expertise in EIB projects; it is also responsible for assessing the impact of EIB 
activities and the results generated by EIB-financed projects. Her experience includes an adjunct 
professorship in macroeconomics at Bocconi University, Italy, a research role at Banca Commerciale 
Italiana, and a chief economist role for Central and Eastern Europe at UniCredit. She is a member of 
the steering committees of the Vienna Initiative and the CompNet, an alternate member of the board 
of the Joint Vienna Institute, and a member of the boards of The European Money and Finance Forum 
(SUERF) and the Euro 50 Group. She holds an undergraduate degree in economics and a master’s 
degree in economics from Bocconi University, as well as a PhD in economics from the University of 
Ancona in Italy. 


Matt Ripley co-authored Chapter 1 and authored Chapter 3. He is a specialist advisor in impact 
management strategies and systems, working with investors and businesses to “prove and improve” 
social outcomes. He has a wealth of experience in the “profit with purpose” space as a practitioner 
and “thought leader.” He works with The Good Economy, London, United Kingdom, a leading social 
advisory firm, helping clients at every stage of the impact journey to identify and integrate real-world 
impact into decision-making, measurement and reporting. Previously, he ran his own consulting 
company, was a chief technical advisor for small and medium-sized enterprise development at the 
United Nations’ International Labour Organization, and led a UK government-supported agribusiness 
investment fund in Nepal, which pioneered new approaches to developing inclusive rural markets. 


Alexandros (Aleko) Sarris co-edited the book and co-authored Chapter 2. He is an emeritus professor 
in the Department of Economics of the National and Kapodistrian University of Athens, Greece, where 
he lectured from 1982 to 2015. From 2003 to 2010, he was director of the Trade and Markets Division 
of the Food and Agriculture Organization (FAO) of the United Nations. He holds a joint PhD in 
economics and electrical engineering from the Massachusetts Institute of Technology, United States, 
where he also earned a bachelor’s and master’s degree in electrical engineering. From 1976 to 1982, 
he was assistant professor at the Department of Agricultural and Resource Economics at the University 
of California at Berkeley, United States. He has authored or co-authored more than 15 books and 
monographs, and more than 70 journal and book articles on various issues of economic development. 
He has advised many United Nations organisations, as well as national and international agencies, and 
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has field research and policy advice experience in over 20 countries. He was an expert advisor in the 
EIB-GDN programme. 


EIB-GDN research fellows and deep dive authors 


Upon validation of the final version of the deep dives by the expert advisors and the advisory 
committee of the EIB-GDN programme, the participants are formally distinguished as EIB-GDN Fellows 
in Applied Development Finance. As of the end of March 2021, the validation had been completed for 
cycles 1 and 2, and remained pending for cycle 3, as the deep dives were being finalised. 


Cycle 1 


Yesuf Mohammednur Awel (EIB-GDN Fellow in Applied Development Finance, 2019) works at the 
United Nations Economic Commission for Africa (ECA). He is a versatile economist with a research 
focus on development finance, risk and insurance, and impact evaluation of development 
interventions. At the ECA, he works on macroeconomic modelling and policy analysis in Africa. He holds 
a PhD in economics from Maastricht University, Netherlands. 


Charles Yaw Okyere (EIB-GDN Fellow in Applied Development Finance, 2019) is at the Department of 
Agricultural Economics and Agribusiness, College of Basic and Applied Sciences, University of Ghana, 
Accra. He holds a doctor of agricultural sciences (Dr. Agr.) degree from the University of Bonn, 
Germany. His research interests include the application of behavioural, experimental and quasi- 
experimental economic techniques to health, education, agriculture and welfare. He has skills in 
conducting socioeconomic surveys and has been involved in the collection of data on various 
development issues, using both quantitative and qualitative techniques. 


Oluwasola Emmanuel Omoju (EIB-GDN Fellow in Applied Development Finance, 2019) is a research 
fellow at the National Institute for Legislative and Democratic Studies (NILDS), Abuja, Nigeria, where 
he provides research and policy support to the National Assembly. He holds a bachelor’s degree in 
economics from the University of Ilorin, Nigeria, a master’s degree in economics from the University 
of Lagos, Nigeria, and a PhD in applied (energy) economics from Xiamen University, China. His research 
interests include energy economics and policy, public financial management, applied development 
economics, and trade policy with special reference to the African Continental Free Trade Area 
(AfCFTA). 


Mariam Sangaré (EIB-GDN Fellow in Applied Development Finance, 2019) has a PhD in economics from 
the University of Toulouse, France. Her research interests cover low-income people’s demand for 
financial services, impact analysis of microfinance services, mobile banking, and questions around 
financial services supply and demand matching. She is currently an associate professor in economics 
and management at the University of Bamako, Mali. 


Bassirou Sarr (EIB-GDN Fellow in Applied Development Finance, 2019) is a doctoral candidate at the 
Paris School of Economics, France. His research interests are in development, public finance and 
applied economics. He has conducted impact evaluation and experimental economics research in Mali, 
Senegal and Benin. These research projects span areas as diverse as taxation and applied game theory. 
As part of his dissertation on state taxation and redistribution, he was the principal investigator on 
innovative research projects in collaboration with Senegal’s Directorate of Taxes and Domains to 
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address research questions on VAT design, personal income taxes and property tax administration. He 
obtained an MSc degree in analysis and policy economics. He also graduated from Carleton College, 
Minnesota, United States, where he earned a bachelor of mathematics (BMath) degree. 


Guylaine Nouwoue Epse Tchakounte (EIB-GDN Fellow in Applied Development Finance, 2019) is at 
the University of Exeter Tax Administration Research Centre (TARC), United Kingdom. She is a PhD 
applicant with ten years of experience in policy evaluation. Her research lies in the areas of behavioural 
economics, development economics and public finance in Africa. 


Gloria Uwingabiye (EIB-GDN Fellow in Applied Development Finance, 2019) works on the evaluation 
and impact assessment of financial and macroeconomic policies at ICF International. She has extensive 
international experience and has worked in the private, not-for-profit and academic sectors in various 
countries. 


Eleni Yitbarek (EIB-GDN Fellow in Applied Development Finance, 2019) is a senior lecturer at the 
University of Pretoria Economics Department, South Africa. She obtained her PhD in economics and 
an MSc in public policy and human development, specialising in social policy financing, from Maastricht 
University, Netherlands. Her research interests revolve around applied research in poverty dynamics, 
socioeconomic effects of shocks, impact evaluation and gender-based social mobility. 


Cycle 2 


Abdi Yuya Ahmad (EIB-GDN Fellow in Applied Development Finance, 2020) is an assistant professor at 
the Department of Technology and Innovation Management, Adama Science and Technology 
University, Ethiopia. He earned his PhD in innovation economics from Aalborg University, Denmark, 
and his MSc in economic policy analysis from Addis Ababa University, Ethiopia. 


Franklin Amuakwa-Mensah (EIB-GDN Fellow in Applied Development Finance, 2020) is a research 
manager and researcher at the Environment for Development (EfD) unit, University of Gothenburg, 
Sweden. He is also a research fellow on the Transfer project at the United Nations Children’s Fund 
(Unicef) Innocenti Research Centre. He was formerly a postdoctoral researcher at the Department of 
Economics at the Swedish University of Agricultural Sciences (SLU), Uppsala, from which he holds a 
PhD degree in economics. 


Laura Barasa (EIB-GDN Fellow in Applied Development Finance, 2020) holds a PhD in economics from 
Radboud University Nijmegen, Netherlands. She teaches econometrics and economic statistics at the 
School of Economics of the University of Nairobi, Kenya. Her research interests include innovation, 
gender, agriculture and poverty in sub-Saharan Africa. 


Edgar F.A. Cooke (EIB-GDN Fellow in Applied Development Finance, 2020) is a lecturer in the Business 
Administration Department at Ashesi University, Accra, Ghana. He teaches microeconomics, 
macroeconomics, and international trade and policy to undergraduate students. His principal research 
interests are in the areas of poverty, inequality, economic growth and international trade. 


Hamidou Jawara (EIB-GDN Fellow in Applied Development Finance, 2020) is at the University of The 
Gambia, Serekunda. He is a young economist whose principal interest is in development economics. 
His research work has been on diverse areas including financial inclusion, digital finance, food security 
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and nutrition, poverty, health and welfare. He also has consultancy experience with different 
organisations such as the World Bank, International Labour Organization and International Trade 
Centre. 


Gilles Quentin Kane (EIB-GDN Fellow in Applied Development Finance, 2020) is an assistant lecturer 
at the Faculty of Economics and Management of the University of Yaoundé II, Cameroon. He earned a 
PhD in economics (agricultural economics) in 2018 under the collaborative programme of the African 
Economic Research Consortium (AERC). He has over five years of experience in research, lecturing and 
consulting for well-renowned organisations such as the World Bank, the GDN and SNV Netherlands. 
His research interests include food and nutrition security, digital agriculture, food price, impact 
evaluation, gender and poverty analysis. His research has been published in international journals such 
as the Economics Bulletin, the Review of Development Economics, and Poverty and Public Policy. 


Timothy Kinoti (EIB-GDN Fellow in Applied Development Finance, 2020) holds a PhD from the Jomo 
Kenyatta University of Agriculture and Technology, Juja. He is an international development 
professional at the World University Service of Canada (WUSC), whose principal interests are in 
development finance and gender. His work has included diverse areas such as social impact investing, 
impact analysis, gender, digital technologies and poverty. He has experience working in different 
sectors, and is currently working on refugee programmes on education, livelihoods and cash transfers. 
He is also currently involved in studying artificial intelligence in social sciences. 


Sied Hassen Mohamed (EIB-GDN Fellow in Applied Development Finance, 2020) is a lead researcher 
at the Policy Studies Institute (PSI) in Addis Ababa, Ethiopia. He previously worked as a lecturer and 
assistant professor at Mekelle University. His areas of research are in energy and development 
economics. 


Muhammed Abdella Usman (EIB-GDN Fellow in Applied Development Finance, 2020) is a senior 
researcher at the Centre for Development Research (ZEF), University of Bonn. He specialised as a 
developmental and agricultural economist, with a strong background in applied microeconomics and 
strong capabilities in quantitative analysis. He has rich experience in designing, supervising, collecting 
and analysing large-scale household survey data in developing countries. He has been working on a 
range of development issues such as information technology and agricultural markets, food and 
nutrition security, water supply and sanitation, child health and growth, agricultural technology 
adoption and impact evaluation. 


Soazic Elise Wang Sonne (EIB-GDN Fellow in Applied Development Finance, 2020) is an economist 
(young professional) with the Social Sustainability and Inclusion Global Practice of the World Bank 
Group in Washington D.C., United States. She is currently on a development assignment with the 
Health, Nutrition and Population (HNP) unit, supporting projects aiming to strengthen health systems 
in fragile and conflict-affected situations for a better response to the COVID-19 pandemic. 


Cycle 3 


Esther Leah Achandi (EIB-GDN Candidate Fellow in Applied Development Finance) is a postdoctoral 
fellow in gender at the International Livestock Research Institute (ILRI), Addis Ababa, Ethiopia. She 
holds a PhD in economics from the University of Dar es Salaam, Tanzania, and has more than nine years 
of research experience in Eastern Africa covering Uganda, Rwanda, Tanzania, Ethiopia, Madagascar 
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and Burundi, with a focus on agricultural value chains; technologies, gender and consumer food 
preferences; and sexual and reproductive health and rights among vulnerable youths. Her work at the 
ILRI focuses on the contribution of livestock species to women’s empowerment in Uganda, Vietnam, 
Ethiopia and Tanzania. She is a member of the African Women in Agricultural Research and 
Development (AWARD) fellowship alumni; a steering committee member on the International Union 
for the Scientific Study of Population (IUSSP) Panel on Population, Poverty and Inequality (2018-2021); 
and a Pan-African Scientific Research Council inaugural fellow (2021). 


Boaz Anglade (EIB-GDN Candidate Fellow in Applied Development Finance) is a development 
economist presently working as a consultant for the Inter-American Development Bank, Washington 
D.C., United States. He holds a PhD in applied economics from the University of Florida, Gainesville, a 
master’s degree in business economics from the University of South Florida, Tampa, and a bachelor’s 
degree in economics from Florida International University, Miami, all in the United States. He has more 
than seven years of experience in conducting empirical research in the field of applied economics, and 
has offered key input to scholarships in the area of development economics through teaching, training 
and academic publishing. His research interests focus on gender disparities, poverty, and diffusion of 
agricultural technologies in developing countries. 


Jaison Chireshe (EIB-GDN Candidate Fellow in Applied Development Finance) holds a PhD in economics 
from the University of the Western Cape, Cape Town, South Africa, and a master’s degree in 
economics. He is a monitoring, evaluation and research consultant with 12 years of experience in 
designing and implementing monitoring, evaluation and learning systems for food security, water and 
sanitation, and social protection projects in Ghana, Sudan, Zambia and Zimbabwe. His research 
interests are in financial development, health outcomes and climate change, and he has several 
publications in these areas. 


Kebba Jammeh (EIB-GDN Candidate Fellow in Applied Development Finance) is an economics lecturer 
at the University of The Gambia, Serekunda. He works as a consultant with the World Bank and the 
African Development Bank. He holds a PhD in economics from the University of Essex, United Kingdom, 
having previously attended the London School of Economics, Queen Mary University of London and 
the University of The Gambia. 


Mary Gorret Nantongo (EIB-GDN Candidate Fellow in Applied Development Finance) is at the 
Department of Economics, Makerere University Business School, Kampala, Uganda. She holds a PhD in 
development and environment studies and an MSc in development and natural resource economics 
from the Norwegian University of Life Sciences. Her research uses impact evaluation methods and 
governance theory to evaluate the processes, costs and impacts of natural resources, as well as 
climate, environment and energy policies and programmes. 


Jacob Novignon (EIB-GDN Candidate Fellow in Applied Development Finance) is a lecturer at the 
Department of Economics of Kwame Nkrumah University of Science and Technology (KNUST), Kumasi, 
Ghana. His research interest is in the broad field of development microeconomics. 


Sabine Mireille Ntsama Etoundi (EIB-GDN Candidate Fellow in Applied Development Finance) is at the 
University of Yaoundé II, Soa, Cameroon, and the SQuare Foundation. She is a young female economist 
from Cameroon with ten years of experience in policy-oriented and applied research in development 
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economics. Her articles have been published in top journals in development economics (World 
Development and Oxford Economics Papers), focusing on agricultural trade, institutional quality, 
natural resources and education. Her geographical region of expertise is sub-Saharan Africa. She is the 
founding member of the SQuare Foundation, a Cameroon-based think tank that specialises in practical 
research and solutions for long-term economic development. 


Enoch Owusu-Sekyere (EIB-GDN Candidate Fellow in Applied Development Finance) is at the 
Department of Economics of the Swedish University of Agricultural Sciences (SLU), Uppsala, and the 
Department of Agricultural Economics of the University of the Free State, South Africa. He is a 
researcher in the Agricultural and Food Economics group at the SLU and holds a PhD degree in 
agricultural economics. His research focuses on impact assessment of private and government sector 
investment, farm management, animal welfare, technology adoption, consumer behaviour, the 
circular economy, food policy, food security and safety, ecological footprint assessment, econometric 
modelling and agribusiness. 


Matthew Townshend (EIB-GDN Candidate Fellow in Applied Development Finance) is an economist at 
the University of Cape Town, South Africa, who focuses on public finance and policy. He works as a 
consultant, combining his academic training and practical experience to help implement evidence- 
based interventions that optimise development impacts. He has helped develop and review 12 
national policies in Eswatini, Ghana, Nigeria, South Africa, Tajikistan and Uzbekistan. 


Alexander Uchenna Nnamani (EIB-GDN Candidate Fellow in Applied Development Finance) is at the 
Development Strategy Centre, Enugu, Nigeria. He is an economist and a young policy researcher in the 
field of development economics. He has five years of experience working with a Nigerian-based think 
tank, the Development Strategy Centre, and other international organisations like the African 
Economic Research Consortium (AERC), the GDN and the EIB. His recent research covers topics in 
infrastructure finance, trade and regional integration, and capital flows (including remittances and 
foreign investment). 
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